1. 15 Mar, 2018 2 commits
  2. 28 Feb, 2018 1 commit
  3. 27 Feb, 2018 1 commit
    • antirez's avatar
      AOF: fix a bug that may prevent proper fsyncing when fsync=always. · 75987431
      antirez authored
      In case the write handler is already installed, it could happen that we
      serve the reply of a query in the same event loop cycle we received it,
      preventing beforeSleep() from guaranteeing that we do the AOF fsync
      before sending the reply to the client.
      
      The AE_BARRIER mechanism, introduced in a previous commit, prevents this
      problem. This commit makes actual use of this new feature to fix the
      bug.
      75987431
  4. 13 Feb, 2018 1 commit
  5. 18 Jan, 2018 1 commit
    • Guy Benoish's avatar
      Replication buffer fills up on high rate traffic. · fd8efb7c
      Guy Benoish authored
      When feeding the master with a high rate traffic the the slave's feed is much slower.
      This causes the replication buffer to grow (indefinitely) which leads to slave disconnection.
      The problem is that writeToClient() decides to stop writing after NET_MAX_WRITES_PER_EVENT
      writes (In order to be fair to clients).
      We should ignore this when the client is a slave.
      It's better if clients wait longer, the alternative is that the slave has no chance to stay in
      sync in this situation.
      fd8efb7c
  6. 11 Jan, 2018 1 commit
  7. 29 Dec, 2017 2 commits
  8. 06 Dec, 2017 1 commit
    • antirez's avatar
      Change indentation and other minor details of PR #4489. · 522760fa
      antirez authored
      The main change introduced by this commit is pretending that help
      arrays are more text than code, thus indenting them at level 0. This
      improves readability, and is an old practice when defining arrays of
      C strings describing text.
      
      Additionally a few useless return statements are removed, and the HELP
      subcommand capitalized when printed to the user.
      522760fa
  9. 05 Dec, 2017 3 commits
  10. 03 Dec, 2017 1 commit
  11. 01 Dec, 2017 2 commits
  12. 30 Nov, 2017 1 commit
  13. 28 Nov, 2017 1 commit
    • Itamar Haber's avatar
      Standardizes the 'help' subcommand · 59d52f7f
      Itamar Haber authored
      This adds a new `addReplyHelp` helper that's used by commands
      when returning a help text. The following commands have been
      touched: DEBUG, OBJECT, COMMAND, PUBSUB, SCRIPT and SLOWLOG.
      
      WIP
      
      Fix entry command table entry for OBJECT for HELP option.
      
      After #4472 the command may have just 2 arguments.
      
      Improve OBJECT HELP descriptions.
      
      See #4472.
      
      WIP 2
      
      WIP 3
      59d52f7f
  14. 11 Jul, 2017 1 commit
  15. 05 Jul, 2017 1 commit
  16. 04 Jul, 2017 1 commit
    • antirez's avatar
      Add symmetrical assertion to track c->reply_buffer infinite growth. · eddd8d34
      antirez authored
      Redis clients need to have an instantaneous idea of the amount of memory
      they are consuming (if the number is not exact should at least be
      proportional to the actual memory usage). We do that adding and
      subtracting the SDS length when pushing / popping from the client->reply
      list. However it is quite simple to add bugs in such a setup, by not
      taking the objects in the list and the count in sync. For such reason,
      Redis has an assertion to track counts near 2^64: those are always the
      result of the counter wrapping around because we subtract more than we
      add. This commit adds the symmetrical assertion: when the list is empty
      since we sent everything, the reply_bytes count should be zero. Thanks
      to the new assertion it should be simple to also detect the other
      problem, where the count slowly increases because of over-counting.
      The assertion adds a conditional in the code that sends the buffer to
      the socket but should not create any measurable performance slowdown,
      listLength() just accesses a structure field, and this code path is
      totally dominated by write(2).
      
      Related to #4100.
      eddd8d34
  17. 09 May, 2017 1 commit
  18. 19 Apr, 2017 1 commit
    • antirez's avatar
      Fix PSYNC2 incomplete command bug as described in #3899. · 22be435e
      antirez authored
      This bug was discovered by @kevinmcgehee and constituted a major hidden
      bug in the PSYNC2 implementation, caused by the propagation from the
      master of incomplete commands to slaves.
      
      The bug had several results:
      
      1. Borrowing from Kevin text in the issue: "Given that slaves blindly
      copy over their master's input into their own replication backlog over
      successive read syscalls, it's possible that with large commands or
      small TCP buffers, partial commands are present in this buffer. If the
      master were to fail before successfully propagating the entire command
      to a slave, the slaves will never execute the partial command (since the
      client is invalidated) but will copy it to replication backlog which may
      relay those invalid bytes to its slaves on PSYNC2, corrupting the
      backlog and possibly other valid commands that follow the failover.
      Simple command boundaries aren't sufficient to capture this, either,
      because in the case of a MULTI/EXEC block, if the master successfully
      propagates a subset of the commands but not the EXEC, then the
      transaction in the backlog becomes corrupt and could corrupt other
      slaves that consume this data."
      
      2. As identified by @yangsiran later, there is another effect of the
      bug. For the same mechanism of the first problem, a slave having another
      slave, could receive a full resynchronization request with an already
      half-applied command in the backlog. Once the RDB is ready, it will be
      sent to the slave, and the replication will continue sending to the
      sub-slave the other half of the command, which is not valid.
      
      The fix, designed by @yangsiran and @antirez, and implemented by
      @antirez, uses a secondary buffer in order to feed the sub-masters and
      update the replication backlog and offsets, only when a given part of
      the query buffer is actually *applied* to the state of the instance,
      that is, when the command gets processed and the command is not pending
      in the Redis transaction buffer because of CLIENT_MULTI state.
      
      Given that now the backlog and offsets representation are in agreement
      with the actual processed commands, both issue 1 and 2 should no longer
      be possible.
      
      Thanks to @kevinmcgehee, @yangsiran and @oranagra for their work in
      identifying and designing a fix for this problem.
      22be435e
  19. 12 Apr, 2017 1 commit
  20. 23 Feb, 2017 2 commits
  21. 10 Feb, 2017 1 commit
  22. 28 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: stop sending newlines to sub-slaves when master is down. · eab865a0
      antirez authored
      This actually includes two changes:
      
      1) No newlines to take the master-slave link up when the upstream master
      is down. Doing this is dangerous because the sub-slave often is received
      replication protocol for an half-command, so can't receive newlines
      without desyncing the replication link, even with the code in order to
      cancel out the bytes that PSYNC2 was using. Moreover this is probably
      also not needed/sane, because anyway the slave can keep serving
      requests, and because if it's configured to don't serve stale data, it's
      a good idea, actually, to break the link.
      
      2) When a +CONTINUE with a different ID is received, we now break
      connection with the sub-slaves: they need to be notified as well. This
      was part of the original specification but for some reason it was not
      implemented in the code, and was alter found as a PSYNC2 bug in the
      integration testing.
      eab865a0
  23. 25 Nov, 2016 1 commit
  24. 09 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  25. 03 Aug, 2016 1 commit
    • antirez's avatar
      Security: Cross Protocol Scripting protection. · a81a92ca
      antirez authored
      This is an attempt at mitigating problems due to cross protocol
      scripting, an attack targeting services using line oriented protocols
      like Redis that can accept HTTP requests as valid protocol, by
      discarding the invalid parts and accepting the payloads sent, for
      example, via a POST request.
      
      For this to be effective, when we detect POST and Host: and terminate
      the connection asynchronously, the networking code was modified in order
      to never process further input. It was later verified that in a
      pipelined request containing a POST command, the successive commands are
      not executed.
      a81a92ca
  26. 27 Jul, 2016 1 commit
    • antirez's avatar
      Ability of slave to announce arbitrary ip/port to master. · 55385f99
      antirez authored
      This feature is useful, especially in deployments using Sentinel in
      order to setup Redis HA, where the slave is executed with NAT or port
      forwarding, so that the auto-detected port/ip addresses, as listed in
      the "INFO replication" output of the master, or as provided by the
      "ROLE" command, don't match the real addresses at which the slave is
      reachable for connections.
      55385f99
  27. 23 May, 2016 1 commit
  28. 10 May, 2016 1 commit
  29. 25 Apr, 2016 2 commits
  30. 18 Feb, 2016 1 commit
  31. 20 Jan, 2016 1 commit
  32. 07 Jan, 2016 2 commits