• Viktor Söderqvist's avatar
    Wait for replicas when shutting down (#9872) · 45a155bd
    Viktor Söderqvist authored
    
    
    To avoid data loss, this commit adds a grace period for lagging replicas to
    catch up the replication offset.
    
    Done:
    
    * Wait for replicas when shutdown is triggered by SIGTERM and SIGINT.
    
    * Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new
      blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients
      to call SHUTDOWN in parallel.
      Note that they don't expect a response unless an error happens and shutdown is aborted.
    
    * Log warning for each replica lagging behind when finishing shutdown.
    
    * CLIENT_PAUSE_WRITE while waiting for replicas.
    
    * Configurable grace period 'shutdown-timeout' in seconds (default 10).
    
    * New flags for the SHUTDOWN command:
    
        - NOW disables the grace period for lagging replicas.
    
        - FORCE ignores errors writing the RDB or AOF files which would normally
          prevent a shutdown.
    
        - ABORT cancels ongoing shutdown. Can't be combined with other flags.
    
    * New field in the output of the INFO command: 'shutdown_in_milliseconds'. The
      value is the remaining maximum time to wait for lagging replicas before
      finishing the shutdown. This field is present in the Server section **only**
      during shutdown.
    
    Not directly related:
    
    * When shutting down, if there is an AOF saving child, it is killed **even** if AOF
      is disabled. This can happen if BGREWRITEAOF is used when AOF is off.
    
    * Client pause now has end time and type (WRITE or ALL) per purpose. The
      different pause purposes are *CLIENT PAUSE command*, *failover* and
      *shutdown*. If clients are unpaused for one purpose, it doesn't affect client
      pause for other purposes. For example, the CLIENT UNPAUSE command doesn't
      affect client pause initiated by the failover or shutdown procedures. A completed
      failover or a failed shutdown doesn't unpause clients paused by the CLIENT
      PAUSE command.
    
    Notes:
    
    * DEBUG RESTART doesn't wait for replicas.
    
    * We already have a warning logged when a replica disconnects. This means that
      if any replica connection is lost during the shutdown, it is either logged as
      disconnected or as lagging at the time of exit.
    Co-authored-by: default avatarOran Agra <oran@redislabs.com>
    45a155bd
server.tcl 23.2 KB