1. 30 Jun, 2018 1 commit
  2. 26 Jun, 2018 4 commits
  3. 06 Jun, 2018 1 commit
  4. 16 Jan, 2018 2 commits
  5. 05 Dec, 2017 1 commit
    • antirez's avatar
      add linkClient(): adds the client and caches the list node. · 62a4b817
      antirez authored
      We have this operation in two places: when caching the master and
      when linking a new client after the client creation. By having an API
      for this we avoid incurring in errors when modifying one of the two
      places forgetting the other. The function is also a good place where to
      document why we cache the linked list node.
      
      Related to #4497 and #4210.
      62a4b817
  6. 30 Nov, 2017 1 commit
  7. 24 Nov, 2017 1 commit
  8. 01 Nov, 2017 1 commit
    • zhaozhao.zz's avatar
      PSYNC2: safe free backlog when reach the time limit · 6ddf0ea2
      zhaozhao.zz authored
      When we free the backlog, we should use a new
      replication ID and clear the ID2. Since without
      backlog we can not increment master_repl_offset
      even do write commands, that may lead to inconsistency
      when we try to connect a "slave-before" master
      (if this master is our slave before, our replid
      equals the master's replid2). As the master have our
      history, so we can match the master's replid2 and
      second_replid_offset, that make partial sync work,
      but the data is inconsistent.
      6ddf0ea2
  9. 20 Sep, 2017 2 commits
    • antirez's avatar
      PSYNC2: More refinements related to #4316. · bb3b5ddd
      antirez authored
      bb3b5ddd
    • zhaozhao.zz's avatar
      PSYNC2: make persisiting replication info more solid · b541ccef
      zhaozhao.zz authored
      This commit is a reinforcement of commit c1c99e9f.
      
      1. Replication information can be stored when the RDB file is
      generated by a mater using server.slaveseldb when server.repl_backlog
      is not NULL, or set repl_stream_db be -1. That's safe, because
      NULL server.repl_backlog will trigger full synchronization,
      then master will send SELECT command to replicaiton stream.
      2. Only do rdbSave* when rsiptr is not NULL,
      if we do rdbSave* without rdbSaveInfo, slave will miss repl-stream-db.
      3. Save the replication informations also in the case of
      SAVE command, FLUSHALL command and DEBUG reload.
      b541ccef
  10. 19 Sep, 2017 2 commits
    • antirez's avatar
      PSYNC2: Fix the way replication info is saved/loaded from RDB. · c1c99e9f
      antirez authored
      This commit attempts to fix a number of bugs reported in #4316.
      They are related to the way replication info like replication ID,
      offsets, and currently selected DB in the master client, are stored
      and loaded by Redis. In order to avoid inconsistencies the changes in
      this commit try to enforce that:
      
      1. Replication information are only stored when the RDB file is
      generated by a slave that has a valid 'master' client, so that we can
      always extract the currently selected DB.
      2. When replication informations are persisted in the RDB file, all the
      info for a successful PSYNC or nothing is persisted.
      3. The RDB replication informations are only loaded if the instance is
      configured as a slave, otherwise a master can start with IDs that relate
      to a different history of the data set, and stil retain such IDs in the
      future while receiving unrelated writes.
      c1c99e9f
    • antirez's avatar
      PSYNC2: Create backlog on slave partial sync as well. · b75ae0bb
      antirez authored
      A slave may be started with an RDB file able to provide enough slave to
      perform a successful partial SYNC with its master. However in such a
      case, how outlined in issue #4268, the slave backlog will not be
      started, since it was only initialized on full syncs attempts. This
      creates different problems with successive PSYNC attempts that will
      always result in full synchronizations.
      
      Thanks to @fdingiit for discovering the issue.
      b75ae0bb
  11. 31 Aug, 2017 1 commit
  12. 27 Apr, 2017 1 commit
    • antirez's avatar
      PSYNC2: fix master cleanup when caching it. · 469d6e2b
      antirez authored
      The master client cleanup was incomplete: resetClient() was missing and
      the output buffer of the client was not reset, so pending commands
      related to the previous connection could be still sent.
      
      The first problem caused the client argument vector to be, at times,
      half populated, so that when the correct replication stream arrived the
      protcol got mixed to the arugments creating invalid commands that nobody
      called.
      
      Thanks to @yangsiran for also investigating this problem, after
      already providing important design / implementation hints for the
      original PSYNC2 issues (see referenced Github issue).
      
      Note that this commit adds a new function to the list library of Redis
      in order to be able to reset a list without destroying it.
      
      Related to issue #3899.
      469d6e2b
  13. 19 Apr, 2017 2 commits
    • antirez's avatar
      PSYNC2: discard pending transactions from cached master. · 189a12af
      antirez authored
      During the review of the fix for #3899, @yangsiran identified an
      implementation bug: given that the offset is now relative to the applied
      part of the replication log, when we cache a master, the successive
      PSYNC2 request will be made in order to *include* the transaction that
      was not completely processed. This means that we need to discard any
      pending transaction from our replication buffer: it will be re-executed.
      189a12af
    • antirez's avatar
      Fix PSYNC2 incomplete command bug as described in #3899. · 22be435e
      antirez authored
      This bug was discovered by @kevinmcgehee and constituted a major hidden
      bug in the PSYNC2 implementation, caused by the propagation from the
      master of incomplete commands to slaves.
      
      The bug had several results:
      
      1. Borrowing from Kevin text in the issue: "Given that slaves blindly
      copy over their master's input into their own replication backlog over
      successive read syscalls, it's possible that with large commands or
      small TCP buffers, partial commands are present in this buffer. If the
      master were to fail before successfully propagating the entire command
      to a slave, the slaves will never execute the partial command (since the
      client is invalidated) but will copy it to replication backlog which may
      relay those invalid bytes to its slaves on PSYNC2, corrupting the
      backlog and possibly other valid commands that follow the failover.
      Simple command boundaries aren't sufficient to capture this, either,
      because in the case of a MULTI/EXEC block, if the master successfully
      propagates a subset of the commands but not the EXEC, then the
      transaction in the backlog becomes corrupt and could corrupt other
      slaves that consume this data."
      
      2. As identified by @yangsiran later, there is another effect of the
      bug. For the same mechanism of the first problem, a slave having another
      slave, could receive a full resynchronization request with an already
      half-applied command in the backlog. Once the RDB is ready, it will be
      sent to the slave, and the replication will continue sending to the
      sub-slave the other half of the command, which is not valid.
      
      The fix, designed by @yangsiran and @antirez, and implemented by
      @antirez, uses a secondary buffer in order to feed the sub-masters and
      update the replication backlog and offsets, only when a given part of
      the query buffer is actually *applied* to the state of the instance,
      that is, when the command gets processed and the command is not pending
      in the Redis transaction buffer because of CLIENT_MULTI state.
      
      Given that now the backlog and offsets representation are in agreement
      with the actual processed commands, both issue 1 and 2 should no longer
      be possible.
      
      Thanks to @kevinmcgehee, @yangsiran and @oranagra for their work in
      identifying and designing a fix for this problem.
      22be435e
  14. 12 Apr, 2017 1 commit
  15. 20 Feb, 2017 1 commit
  16. 16 Dec, 2016 1 commit
    • antirez's avatar
      Hopefully improve code comments for issue #3616. · 8e390a62
      antirez authored
      This commit also contains other changes in order to conform the code to
      the Redis core style, specifically 80 chars max per line, smart
      conditionals in the same line:
      
          if (that) do_this();
      8e390a62
  17. 02 Dec, 2016 1 commit
  18. 29 Nov, 2016 1 commit
  19. 28 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: stop sending newlines to sub-slaves when master is down. · eab865a0
      antirez authored
      This actually includes two changes:
      
      1) No newlines to take the master-slave link up when the upstream master
      is down. Doing this is dangerous because the sub-slave often is received
      replication protocol for an half-command, so can't receive newlines
      without desyncing the replication link, even with the code in order to
      cancel out the bytes that PSYNC2 was using. Moreover this is probably
      also not needed/sane, because anyway the slave can keep serving
      requests, and because if it's configured to don't serve stale data, it's
      a good idea, actually, to break the link.
      
      2) When a +CONTINUE with a different ID is received, we now break
      connection with the sub-slaves: they need to be notified as well. This
      was part of the original specification but for some reason it was not
      implemented in the code, and was alter found as a PSYNC2 bug in the
      integration testing.
      eab865a0
  20. 24 Nov, 2016 1 commit
  21. 23 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: bugfixing pre release. · 5b7d42ff
      antirez authored
      1. Master replication offset was cleared after switching configuration
      to some other slave, since it was assumed you can't PSYNC after a
      switch. Note the case anymore and when we successfully PSYNC we need to
      have our offset untouched.
      
      2. Secondary replication ID was not reset to "000..." pattern at
      startup.
      
      3. Master in error state replying -LOADING or other transient errors
      forced the slave to discard the cached master and full resync. This is
      now fixed.
      
      4. Better logging of what's happening on failed PSYNCs.
      5b7d42ff
  22. 16 Nov, 2016 2 commits
  23. 10 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: Save replication ID/offset on RDB file. · 28c96d73
      antirez authored
      This means that stopping a slave and restarting it will still make it
      able to PSYNC with the master. Moreover the master itself will retain
      its ID/offset, in case it gets turned into a slave, or if a slave will
      try to PSYNC with it with an exactly updated offset (otherwise there is
      no backlog).
      
      This change was possible thanks to PSYNC v2 that makes saving the current
      replication state much simpler.
      28c96d73
  24. 09 Nov, 2016 2 commits
    • antirez's avatar
      PSYNC2: Wrap debugging code with if(0) · 4e5e366e
      antirez authored
      4e5e366e
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  25. 24 Sep, 2016 1 commit
  26. 02 Aug, 2016 1 commit
  27. 27 Jul, 2016 1 commit
    • antirez's avatar
      Ability of slave to announce arbitrary ip/port to master. · 55385f99
      antirez authored
      This feature is useful, especially in deployments using Sentinel in
      order to setup Redis HA, where the slave is executed with NAT or port
      forwarding, so that the auto-detected port/ip addresses, as listed in
      the "INFO replication" output of the master, or as provided by the
      "ROLE" command, don't match the real addresses at which the slave is
      reachable for connections.
      55385f99
  28. 22 Jul, 2016 1 commit
    • antirez's avatar
      Replication: when possible start RDB saving ASAP. · 03f5b508
      antirez authored
      In a previous commit the replication code was changed in order to
      centralize the BGSAVE for replication trigger in replicationCron(),
      however after further testings, the 1 second delay imposed by this
      change is not acceptable.
      
      So now the BGSAVE is only delayed if the AOF rewriting process is
      active. However past comments made sure that replicationCron() is always
      able to trigger the BGSAVE when needed, making the code generally more
      robust.
      
      The new code is more similar to the initial @oranagra patch where the
      BGSAVE was delayed only if an AOF rewrite was in progress.
      
      Trivia: delaying the BGSAVE uncovered a minor Sentinel issue that is now
      fixed.
      03f5b508
  29. 21 Jul, 2016 1 commit
  30. 03 Dec, 2015 1 commit
    • antirez's avatar
      Centralize slave replication handshake aborting. · acc2336f
      antirez authored
      Now we have a single function to call in any state of the slave
      handshake, instead of using different functions for different states
      which is error prone. Change performed in the context of issue #2479 but
      does not fix it, since should be functionally identical to the past.
      Just an attempt to make replication.c simpler to follow.
      acc2336f
  31. 15 Oct, 2015 1 commit