1. 24 Nov, 2016 1 commit
  2. 23 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: bugfixing pre release. · 5b7d42ff
      antirez authored
      1. Master replication offset was cleared after switching configuration
      to some other slave, since it was assumed you can't PSYNC after a
      switch. Note the case anymore and when we successfully PSYNC we need to
      have our offset untouched.
      
      2. Secondary replication ID was not reset to "000..." pattern at
      startup.
      
      3. Master in error state replying -LOADING or other transient errors
      forced the slave to discard the cached master and full resync. This is
      now fixed.
      
      4. Better logging of what's happening on failed PSYNCs.
      5b7d42ff
  3. 16 Nov, 2016 1 commit
  4. 10 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: Save replication ID/offset on RDB file. · 28c96d73
      antirez authored
      This means that stopping a slave and restarting it will still make it
      able to PSYNC with the master. Moreover the master itself will retain
      its ID/offset, in case it gets turned into a slave, or if a slave will
      try to PSYNC with it with an exactly updated offset (otherwise there is
      no backlog).
      
      This change was possible thanks to PSYNC v2 that makes saving the current
      replication state much simpler.
      28c96d73
  5. 09 Nov, 2016 2 commits
    • antirez's avatar
      PSYNC2: Wrap debugging code with if(0) · 4e5e366e
      antirez authored
      4e5e366e
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  6. 24 Sep, 2016 1 commit
  7. 02 Aug, 2016 1 commit
  8. 27 Jul, 2016 1 commit
    • antirez's avatar
      Ability of slave to announce arbitrary ip/port to master. · 55385f99
      antirez authored
      This feature is useful, especially in deployments using Sentinel in
      order to setup Redis HA, where the slave is executed with NAT or port
      forwarding, so that the auto-detected port/ip addresses, as listed in
      the "INFO replication" output of the master, or as provided by the
      "ROLE" command, don't match the real addresses at which the slave is
      reachable for connections.
      55385f99
  9. 22 Jul, 2016 1 commit
    • antirez's avatar
      Replication: when possible start RDB saving ASAP. · 03f5b508
      antirez authored
      In a previous commit the replication code was changed in order to
      centralize the BGSAVE for replication trigger in replicationCron(),
      however after further testings, the 1 second delay imposed by this
      change is not acceptable.
      
      So now the BGSAVE is only delayed if the AOF rewriting process is
      active. However past comments made sure that replicationCron() is always
      able to trigger the BGSAVE when needed, making the code generally more
      robust.
      
      The new code is more similar to the initial @oranagra patch where the
      BGSAVE was delayed only if an AOF rewrite was in progress.
      
      Trivia: delaying the BGSAVE uncovered a minor Sentinel issue that is now
      fixed.
      03f5b508
  10. 21 Jul, 2016 1 commit
  11. 03 Dec, 2015 1 commit
    • antirez's avatar
      Centralize slave replication handshake aborting. · acc2336f
      antirez authored
      Now we have a single function to call in any state of the slave
      handshake, instead of using different functions for different states
      which is error prone. Change performed in the context of issue #2479 but
      does not fix it, since should be functionally identical to the past.
      Just an attempt to make replication.c simpler to follow.
      acc2336f
  12. 15 Oct, 2015 1 commit
  13. 02 Oct, 2015 1 commit
  14. 01 Oct, 2015 1 commit
  15. 30 Sep, 2015 3 commits
  16. 21 Aug, 2015 1 commit
  17. 20 Aug, 2015 1 commit
    • antirez's avatar
      startBgsaveForReplication(): handle waiting slaves state change. · f18e5b63
      antirez authored
      Before this commit, after triggering a BGSAVE it was up to the caller of
      startBgsavForReplication() to handle slaves in WAIT_BGSAVE_START in
      order to update them accordingly. However when the replication target is
      the socket, this is not possible since the process of updating the
      slaves and sending the FULLRESYNC reply must be coupled with the process
      of starting an RDB save (the reason is, we need to send the FULLSYNC
      command and spawn a child that will start to send RDB data to the slaves
      ASAP).
      
      This commit moves the responsibility of handling slaves in
      WAIT_BGSAVE_START to startBgsavForReplication() so that for both
      diskless and disk-based replication we have the same chain of
      responsiblity. In order accomodate such change, the syncCommand() also
      needs to put the client in the slave list ASAP (just after the initial
      checks) and not at the end, so that startBgsavForReplication() can find
      the new slave alrady in the list.
      
      Another related change is what happens if the BGSAVE fails because of
      fork() or other errors: we now remove the slave from the list of slaves
      and send an error, scheduling the slave connection to be terminated.
      
      As a side effect of this change the following errors found by
      Oran Agra are fixed (thanks!):
      
      1. rdbSaveToSlavesSockets() on failed fork will get the slaves cleaned
      up, otherwise they remain in a wrong state forever since we setup them
      for full resync before actually trying to fork.
      
      2. updateSlavesWaitingBgsave() with replication target set as "socket"
      was broken since the function changed the slaves state from
      WAIT_BGSAVE_START to WAIT_BGSAVE_END via
      replicationSetupSlaveForFullResync(), so later rdbSaveToSlavesSockets()
      will not find any slave in the right state (WAIT_BGSAVE_START) to feed.
      f18e5b63
  18. 07 Aug, 2015 1 commit
    • antirez's avatar
      slaveTryPartialResynchronization and syncWithMaster: better synergy. · bea12591
      antirez authored
      It is simpler if removing the read event handler from the FD is up to
      slaveTryPartialResynchronization, after all it is only called in the
      context of syncWithMaster.
      
      This commit also makes sure that on error all the event handlers are
      removed from the socket before closing it.
      bea12591
  19. 06 Aug, 2015 3 commits
    • antirez's avatar
      syncWithMaster(): non blocking state machine. · 88c716a0
      antirez authored
      88c716a0
    • antirez's avatar
      ce5761e0
    • antirez's avatar
      Replication: add REPLCONF CAPA EOF support. · 3e6d4d59
      antirez authored
      Add the concept of slaves capabilities to Redis, the slave now presents
      to the Redis master with a set of capabilities in the form:
      
          REPLCONF capa SOMECAPA capa OTHERCAPA ...
      
      This has the effect of setting slave->slave_capa with the corresponding
      SLAVE_CAPA macros that the master can test later to understand if it
      the slave will understand certain formats and protocols of the
      replication process. This makes it much simpler to introduce new
      replication capabilities in the future in a way that don't break old
      slaves or masters.
      
      This patch was designed and implemented together with Oran Agra
      (@oranagra).
      3e6d4d59
  20. 05 Aug, 2015 4 commits
    • antirez's avatar
      Fix replication slave pings period. · 55ba7727
      antirez authored
      For PINGs we use the period configured by the user, but for the newlines
      of slaves waiting for an RDB to be created (including slaves waiting for
      the FULLRESYNC reply) we need to ping with frequency of 1 second, since
      the timeout is fixed and needs to be refreshed.
      55ba7727
    • antirez's avatar
      Make sure we re-emit SELECT after each new slave full sync setup. · 15de6b10
      antirez authored
      In previous commits we moved the FULLRESYNC to the moment we start the
      BGSAVE, so that the offset we provide is the right one. However this
      also means that we need to re-emit the SELECT statement every time a new
      slave starts to accumulate the changes.
      
      To obtian this effect in a more clean way, the function that sends the
      FULLRESYNC reply was overloaded with a more important role of also doing
      this and chanigng the slave state. So it was renamed to
      replicationSetupSlaveForFullResync() to better reflect what it does now.
      15de6b10
    • antirez's avatar
      a5a06a8e
    • antirez's avatar
      syncCommand() comments improved. · 62b5c60e
      antirez authored
      62b5c60e
  21. 04 Aug, 2015 1 commit
    • antirez's avatar
      PSYNC initial offset fix. · 292fec05
      antirez authored
      This commit attempts to fix a bug involving PSYNC and diskless
      replication (currently experimental) found by Yuval Inbar from Redis Labs
      and that was later found to have even more far reaching effects (the bug also
      exists when diskstore is off).
      
      The gist of the bug is that, a Redis master replies with +FULLRESYNC to
      a PSYNC attempt that fails and requires a full resynchronization.
      However, the baseline offset sent along with FULLRESYNC was always the
      current master replication offset. This is not ok, because there are
      many reasosn that may delay the RDB file creation. And... guess what,
      the master offset we communicate must be the one of the time the RDB
      was created. So for example:
      
      1) When the BGSAVE for replication is delayed since there is one
         already but is not good for replication.
      2) When the BGSAVE is not needed as we attach one currently ongoing.
      3) When because of diskless replication the BGSAVE is delayed.
      
      In all the above cases the PSYNC reply is wrong and the slave may
      reconnect later claiming to need a wrong offset: this may cause
      data curruption later.
      292fec05
  22. 28 Jul, 2015 2 commits
    • antirez's avatar
      Force slaves to resync after unsuccessful PSYNC. · c1e94b6b
      antirez authored
      Using chained replication where C is slave of B which is in turn slave of
      A, if B reconnects the replication link with A but discovers it is no
      longer possible to PSYNC, slaves of B must be disconnected and PSYNC
      not allowed, since the new B dataset may be completely different after
      the synchronization with the master.
      
      Note that there are varius semantical differences in the way this is
      handled now compared to the past. In the past the semantics was:
      
      1. When a slave lost connection with its master, disconnected the chained
      slaves ASAP. Which is not needed since after a successful PSYNC with the
      master, the slaves can continue and don't need to resync in turn.
      
      2. However after a failed PSYNC the replication backlog was not reset, so a
      slave was able to PSYNC successfully even if the instance did a full
      sync with its master, containing now an entirely different data set.
      
      Now instead chained slaves are not disconnected when the slave lose the
      connection with its master, but only when it is forced to full SYNC with
      its master. This means that if the slave having chained slaves does a
      successful PSYNC all its slaves can continue without troubles.
      
      See issue #2694 for more details.
      c1e94b6b
    • antirez's avatar
  23. 27 Jul, 2015 1 commit
  24. 26 Jul, 2015 6 commits
  25. 11 Jun, 2015 1 commit
    • antirez's avatar
      Use best effort address binding to connect to the master · 8366907b
      antirez authored
      We usually want to reach the master using the address of the interface
      Redis is bound to (via the "bind" config option). That's useful since
      the master will get (and publish) the slave address getting the peer
      name of the incoming socket connection from the slave.
      
      However, when this is not possible, for example because the slave is
      bound to the loopback interface but repliaces from a master accessed via
      an external interface, we want to still connect with the master even
      from a different interface: in this case it is not really important that
      the master will provide any other address, while it is vital to be able
      to replicate correctly.
      
      Related to issues #2609 and #2612.
      8366907b
  26. 01 Apr, 2015 1 commit
    • antirez's avatar
      Net: improve prepareClientToWrite() error handling and comments. · 6c60526d
      antirez authored
      When we fail to setup the write handler it does not make sense to take
      the client around, it is missing writes: whatever is a client or a slave
      anyway the connection should terminated ASAP.
      
      Moreover what the function does exactly with its return value, and in
      which case the write handler is installed on the socket, was not clear,
      so the functions comment are improved to make the goals of the function
      more obvious.
      
      Also related to #2485.
      6c60526d