1. 17 Jul, 2019 1 commit
    • Oran Agra's avatar
      Module API for Forking · 56258c6b
      Oran Agra authored
      * create module API for forking child processes.
      * refactor duplicate code around creating and tracking forks by AOF and RDB.
      * child processes listen to SIGUSR1 and dies exitFromChild in order to
        eliminate a valgrind warning of unhandled signal.
      * note that BGSAVE error reply has changed.
      
      valgrind error is:
        Process terminating with default action of signal 10 (SIGUSR1)
      56258c6b
  2. 08 Jul, 2019 1 commit
    • Oran Agra's avatar
      diskless replication on slave side (don't store rdb to file), plus some other related fixes · 2de544cf
      Oran Agra authored
      The implementation of the diskless replication was currently diskless only on the master side.
      The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.
      
      This commit adds two modes to load rdb directly from socket:
      1) when-empty
      2) using "swapdb"
      the third mode of using diskless slave by flushdb is risky and currently not included.
      
      other changes:
      --------------
      distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
      succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
      also a CONFIG GET and INFO during rdb loading would have lied
      
      When loading rdb from the network, don't kill the server on short read (that can be a network error)
      
      Fix rdb check when performed on preamble AOF
      
      tests:
      run replication tests for diskless slave too
      make replication test a bit more aggressive
      Add test for diskless load swapdb
      2de544cf
  3. 29 Apr, 2019 1 commit
  4. 15 Mar, 2019 1 commit
  5. 12 Mar, 2019 1 commit
  6. 21 Jan, 2019 4 commits
  7. 14 Jan, 2019 1 commit
  8. 16 Oct, 2018 2 commits
  9. 09 Oct, 2018 2 commits
  10. 03 Aug, 2018 2 commits
  11. 16 Jul, 2018 1 commit
    • Oran Agra's avatar
      slave buffers were wasteful and incorrectly counted causing eviction · bf680b6f
      Oran Agra authored
      A) slave buffers didn't count internal fragmentation and sds unused space,
         this caused them to induce eviction although we didn't mean for it.
      
      B) slave buffers were consuming about twice the memory of what they actually needed.
      - this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
        but networking.c not storing more than 16k (partially fixed recently in 237a38737).
      - besides it wasn't able to store half of the new string into one buffer and the
        other half into the next (so the above mentioned fix helped mainly for small items).
      - lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
        consumed but not used.
      
      C) inefficient performance due to starting from a small string and reallocing many times.
      
      what i changed:
      - creating dedicated buffers for reply list, counting their size with zmalloc_size
      - when creating a new reply node from, preallocate it to at least 16k.
      - when appending a new reply to the buffer, first fill all the unused space of the
        previous node before starting a new one.
      
      other changes:
      - expose mem_not_counted_for_evict info field for the benefit of the test suite
      - add a test to make sure slave buffers are counted correctly and that they don't cause eviction
      bf680b6f
  12. 03 Jul, 2018 1 commit
  13. 19 Jun, 2018 2 commits
    • antirez's avatar
      AOF: remove no longer used variable "now". · 333c98c4
      antirez authored
      333c98c4
    • antirez's avatar
      Remove AOF optimization to skip expired keys. · ba92b517
      antirez authored
      Basically we cannot be sure that if the key is expired while writing the
      AOF, the main thread will surely find the key expired. There are
      possible race conditions like the moment at which the "now" is sampled,
      and the fact that time may jump backward.
      
      Think about the following:
      
      SET a 5
      EXPIRE a 1
      
      AOF rewrite starts after about 1 second. The child process finds the key
      expired, while in the main thread instead an INCR command is called
      against the key "a" immediately after a fork, and the scheduler was
      faster to give execution time to the main thread, so "a" is yet not
      expired.
      
      The main thread will generate an INCR a command to the AOF log that will
      be appended to the rewritten AOF file, but that INCR command will target
      a non existin "a" key, so a new non volatile key "a" will be created.
      
      Two observations:
      
      A) In theory by computing "now" before the fork, we should be sure that
      if a key is expired at that time, it will be expired later when the
      main thread will try to access to such key. However this does not take
      into account the fact that the computer time may jump backward.
      
      B) Technically we may still make the process safe by using a monotonic
      time source.
      
      However there were other similar related bugs, and in general the new
      "vision" is that Redis persistence files should represent the memory
      state without trying to be too smart: this makes the design more
      consistent, bugs less likely to arise from complex interactions, and in
      the end what is to fix is the Redis expire process to have less expired
      keys in RAM.
      
      Thanks to Oran Agra and Guy Benoish for writing me an email outlining
      this problem, after they conducted a Redis 5 code review.
      ba92b517
  14. 29 May, 2018 1 commit
    • antirez's avatar
      Don't expire keys while loading RDB from AOF preamble. · 49147f36
      antirez authored
      The AOF tail of a combined RDB+AOF is based on the premise of applying
      the AOF commands to the exact state that there was in the server while
      the RDB was persisted. By expiring keys while loading the RDB file, we
      change the state, so applying the AOF tail later may change the state.
      
      Test case:
      
      * Time1: SET a 10
      * Time2: EXPIREAT a $time5
      * Time3: INCR a
      * Time4: PERSIT A. Start bgrewiteaof with RDB preamble. The value of a is 11 without expire time.
      * Time5: Restart redis from the RDB+AOF: consistency violation.
      
      Thanks to @soloestoy for providing the patch.
      Thanks to @trevor211 for the original issue report and the initial fix.
      
      Check issue #4950 for more info.
      49147f36
  15. 23 Mar, 2018 1 commit
  16. 15 Mar, 2018 1 commit
  17. 13 Feb, 2018 2 commits
    • antirez's avatar
      Make it explicit with a comment why we kill the old AOF rewrite. · c14ba46e
      antirez authored
      See #3858.
      c14ba46e
    • Guy Benoish's avatar
      rewriteAppendOnlyFileBackground() failure fix · f7820067
      Guy Benoish authored
      It is possible to do BGREWRITEAOF even if appendonly=no. This is by design.
      stopAppendonly() didn't turn off aof_rewrite_scheduled (it can be turned on
      again by BGREWRITEAOF even while appendonly is off anyway).
      After configuring `appendonly yes` it will see that the state is AOF_OFF,
      there's no RDB fork, so it will do rewriteAppendOnlyFileBackground() which
      will fail since the aof_child_pid is set (was scheduled and started by cron).
      
      Solution:
      stopAppendonly() will turn off the schedule flag (regardless of who asked for it).
      startAppendonly() will terminate any existing fork and start a new one (so it is the most recent).
      f7820067
  18. 15 Jan, 2018 1 commit
  19. 15 Dec, 2017 1 commit
  20. 14 Dec, 2017 1 commit
  21. 01 Dec, 2017 2 commits
  22. 30 Nov, 2017 2 commits
  23. 22 Jun, 2017 2 commits
  24. 16 Jun, 2017 1 commit
  25. 18 Apr, 2017 1 commit
  26. 23 Feb, 2017 1 commit
  27. 20 Feb, 2017 1 commit
  28. 09 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  29. 06 Oct, 2016 1 commit
    • antirez's avatar
      Module: Ability to get context from IO context. · 152c1b68
      antirez authored
      It was noted by @dvirsky that it is not possible to use string functions
      when writing the AOF file. This sometimes is critical since the command
      rewriting may need to be built in the context of the AOF callback, and
      without access to the context, and the limited types that the AOF
      production functions will accept, this can be an issue.
      
      Moreover there are other needs that we can't anticipate regarding the
      ability to use Redis Modules APIs using the context in order to build
      representations to emit AOF / RDB.
      
      Because of this a new API was added that allows the user to get a
      temporary context from the IO context. The context is auto released
      if obtained when the RDB / AOF callback returns.
      
      Calling multiple time the function to get the context, always returns
      the same one, since it is invalid to have more than a single context.
      152c1b68