1. 10 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: Save replication ID/offset on RDB file. · 28c96d73
      antirez authored
      This means that stopping a slave and restarting it will still make it
      able to PSYNC with the master. Moreover the master itself will retain
      its ID/offset, in case it gets turned into a slave, or if a slave will
      try to PSYNC with it with an exactly updated offset (otherwise there is
      no backlog).
      
      This change was possible thanks to PSYNC v2 that makes saving the current
      replication state much simpler.
      28c96d73
  2. 09 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  3. 14 Oct, 2016 1 commit
    • antirez's avatar
      SWAPDB command. · c7a4e694
      antirez authored
      This new command swaps two Redis databases, so that immediately all the
      clients connected to a given DB will see the data of the other DB, and
      the other way around. Example:
      
          SWAPDB 0 1
      
      This will swap DB 0 with DB 1. All the clients connected with DB 0 will
      immediately see the new data, exactly like all the clients connected
      with DB 1 will see the data that was formerly of DB 0.
      
      MOTIVATION AND HISTORY
      ---
      
      The command was recently demanded by Pedro Melo, but was suggested in
      the past multiple times, and always refused by me.
      
      The reason why it was asked: Imagine you have clients operating in DB 0.
      At the same time, you create a new version of the dataset in DB 1.
      When the new version of the dataset is available, you immediately want
      to swap the two views, so that the clients will transparently use the
      new version of the data. At the same time you'll likely destroy the
      DB 1 dataset (that contains the old data) and start to build a new
      version, to repeat the process.
      
      This is an interesting pattern, but the reason why I always opposed to
      implement this, was that FLUSHDB was a blocking command in Redis before
      Redis 4.0 improvements. Now we have FLUSHDB ASYNC that releases the
      old data in O(1) from the point of view of the client, to reclaim memory
      incrementally in a different thread.
      
      At this point, the pattern can really be supported without latency
      spikes, so I'm providing this implementation for the users to comment.
      In case a very compelling argument will be made against this new command
      it may be removed.
      
      BEHAVIOR WITH BLOCKING OPERATIONS
      ---
      
      If a client is blocking for a list in a given DB, after the swap it will
      still be blocked in the same DB ID, since this is the most logical thing
      to do: if I was blocked for a list push to list "foo", even after the
      swap I want still a LPUSH to reach the key "foo" in the same DB in order
      to unblock.
      
      However an interesting thing happens when a client is, for instance,
      blocked waiting for new elements in list "foo" of DB 0. Then the DB
      0 and 1 are swapped with SWAPDB. However the DB 1 happened to have
      a list called "foo" containing elements. When this happens, this
      implementation can correctly unblock the client.
      
      It is possible that there are subtle corner cases that are not covered
      in the implementation, but since the command is self-contained from the
      POV of the implementation and the Redis core, it cannot cause anything
      bad if not used.
      
      Tests and documentation are yet to be provided.
      c7a4e694
  4. 07 Oct, 2016 1 commit
  5. 06 Oct, 2016 2 commits
    • antirez's avatar
      Fix name of mispelled function. · 799208de
      antirez authored
      799208de
    • antirez's avatar
      Module: Ability to get context from IO context. · 152c1b68
      antirez authored
      It was noted by @dvirsky that it is not possible to use string functions
      when writing the AOF file. This sometimes is critical since the command
      rewriting may need to be built in the context of the AOF callback, and
      without access to the context, and the limited types that the AOF
      production functions will accept, this can be an issue.
      
      Moreover there are other needs that we can't anticipate regarding the
      ability to use Redis Modules APIs using the context in order to build
      representations to emit AOF / RDB.
      
      Because of this a new API was added that allows the user to get a
      temporary context from the IO context. The context is auto released
      if obtained when the RDB / AOF callback returns.
      
      Calling multiple time the function to get the context, always returns
      the same one, since it is invalid to have more than a single context.
      152c1b68
  6. 19 Sep, 2016 1 commit
  7. 16 Sep, 2016 3 commits
  8. 15 Sep, 2016 2 commits
  9. 13 Sep, 2016 2 commits
    • antirez's avatar
      MEMORY OVERHEAD implemented (using Oran Agra initial implementation). · 8c84c962
      antirez authored
      This code was extracted from @oranagra PR #3223 and modified in order
      to provide only certain amounts of information compared to the original
      code. It was also moved from DEBUG to the newly introduced MEMORY
      command. Thanks to Oran for the implementation and the PR.
      
      It implements detailed memory usage stats that can be useful in both
      provisioning and troubleshooting memory usage in Redis.
      8c84c962
    • antirez's avatar
      objectComputeSize(): estimate collections sampling N elements. · 89dec692
      antirez authored
      For most tasks, we need the memory estimation to be O(1) by default.
      This commit also implements an initial MEMORY command.
      Note that objectComputeSize() takes the number of samples to check as
      argument, so MEMORY should be able to get the sample size as option
      to make precision VS CPU tradeoff tunable.
      
      Related to: PR #3223.
      89dec692
  10. 09 Aug, 2016 2 commits
  11. 03 Aug, 2016 1 commit
    • antirez's avatar
      Security: Cross Protocol Scripting protection. · a81a92ca
      antirez authored
      This is an attempt at mitigating problems due to cross protocol
      scripting, an attack targeting services using line oriented protocols
      like Redis that can accept HTTP requests as valid protocol, by
      discarding the invalid parts and accepting the payloads sent, for
      example, via a POST request.
      
      For this to be effective, when we detect POST and Host: and terminate
      the connection asynchronously, the networking code was modified in order
      to never process further input. It was later verified that in a
      pipelined request containing a POST command, the successive commands are
      not executed.
      a81a92ca
  12. 27 Jul, 2016 1 commit
    • antirez's avatar
      Ability of slave to announce arbitrary ip/port to master. · 55385f99
      antirez authored
      This feature is useful, especially in deployments using Sentinel in
      order to setup Redis HA, where the slave is executed with NAT or port
      forwarding, so that the auto-detected port/ip addresses, as listed in
      the "INFO replication" output of the master, or as provided by the
      "ROLE" command, don't match the real addresses at which the slave is
      reachable for connections.
      55385f99
  13. 21 Jul, 2016 1 commit
    • antirez's avatar
      Avoid simultaneous RDB and AOF child process. · 0a628e51
      antirez authored
      This patch, written in collaboration with Oran Agra (@oranagra) is a companion
      to 780a8b1d. Together the two patches should avoid that the AOF and RDB saving
      processes can be spawned at the same time. Previously conditions that
      could lead to two saving processes at the same time were:
      
      1. When AOF is enabled via CONFIG SET and an RDB saving process is
         already active.
      
      2. When the SYNC command decides to start an RDB saving process ASAP in
         order to serve a new slave that cannot partially resynchronize (but
         only if we have a disk target for replication, for diskless
         replication there is not such a problem).
      
      Condition "1" is not very severe but "2" can happen often and is
      definitely good at degrading Redis performances in an unexpected way.
      
      The two commits have the effect of always spawning RDB savings for
      replication in replicationCron() instead of attempting to start an RDB
      save synchronously. Moreover when a BGSAVE or AOF rewrite must be
      performed, they are instead just postponed using flags that will try to
      perform such operations ASAP.
      
      Finally the BGSAVE command was modified in order to accept a SCHEDULE
      option so that if an AOF rewrite is in progress, when this option is
      given, the command no longer returns an error, but instead schedules an
      RDB rewrite operation for when it will be possible to start it.
      0a628e51
  14. 20 Jul, 2016 2 commits
  15. 15 Jul, 2016 1 commit
  16. 13 Jul, 2016 1 commit
    • antirez's avatar
      LRU: Make cross-database choices for eviction. · e423f76e
      antirez authored
      The LRU eviction code used to make local choices: for each DB visited it
      selected the best key to evict. This was repeated for each DB. However
      this means that there could be DBs with very frequently accessed keys
      that are targeted by the LRU algorithm while there were other DBs with
      many better candidates to expire.
      
      This commit attempts to fix this problem for the LRU policy. However the
      TTL policy is still not fixed by this commit. The TTL policy will be
      fixed in a successive commit.
      
      This is an initial (partial because of TTL policy) fix for issue #2647.
      e423f76e
  17. 12 Jul, 2016 2 commits
  18. 06 Jul, 2016 1 commit
  19. 04 Jul, 2016 1 commit
  20. 01 Jul, 2016 1 commit
  21. 22 Jun, 2016 1 commit
  22. 20 Jun, 2016 1 commit
  23. 14 Jun, 2016 1 commit
  24. 13 Jun, 2016 2 commits
  25. 05 Jun, 2016 2 commits
  26. 03 Jun, 2016 1 commit
  27. 01 Jun, 2016 1 commit
  28. 18 May, 2016 1 commit
  29. 10 May, 2016 2 commits