1. 29 May, 2018 2 commits
    • antirez's avatar
      Don't expire keys while loading RDB from AOF preamble. · 49147f36
      antirez authored
      The AOF tail of a combined RDB+AOF is based on the premise of applying
      the AOF commands to the exact state that there was in the server while
      the RDB was persisted. By expiring keys while loading the RDB file, we
      change the state, so applying the AOF tail later may change the state.
      
      Test case:
      
      * Time1: SET a 10
      * Time2: EXPIREAT a $time5
      * Time3: INCR a
      * Time4: PERSIT A. Start bgrewiteaof with RDB preamble. The value of a is 11 without expire time.
      * Time5: Restart redis from the RDB+AOF: consistency violation.
      
      Thanks to @soloestoy for providing the patch.
      Thanks to @trevor211 for the original issue report and the initial fix.
      
      Check issue #4950 for more info.
      49147f36
    • WuYunlong's avatar
      Fix rdb save by allowing dumping of expire keys, so that when · 2a887bd5
      WuYunlong authored
      we add a new slave, and do a failover, eighter by manual or
      not, other local slaves will delete the expired keys properly.
      2a887bd5
  2. 09 May, 2018 2 commits
  3. 08 May, 2018 1 commit
  4. 22 Apr, 2018 1 commit
  5. 16 Mar, 2018 1 commit
  6. 15 Mar, 2018 10 commits
  7. 27 Feb, 2018 1 commit
  8. 29 Dec, 2017 1 commit
    • Oran Agra's avatar
      fix processing of large bulks (above 2GB) · 60a4f12f
      Oran Agra authored
      - protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer
        readQueryFromClient potential overflow
      - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t
      - several places in sds.c that used int for string length or index.
      - bugfix in RM_SaveAuxField (return was 1 or -1 and not length)
      - RM_SaveStringBuffer was limitted to 32bit length
      60a4f12f
  9. 04 Dec, 2017 1 commit
    • antirez's avatar
      Refactoring: improve luaCreateFunction() API. · 60d26acf
      antirez authored
      The function in its initial form, and after the fixes for the PSYNC2
      bugs, required code duplication in multiple spots. This commit modifies
      it in order to always compute the script name independently, and to
      return the SDS of the SHA of the body: this way it can be used in all
      the places, including for SCRIPT LOAD, without duplicating the code to
      create the Lua function name. Note that this requires to re-compute the
      body SHA1 in the case of EVAL seeing a script for the first time, but
      this should not change scripting performance in any way because new
      scripts definition is a rare event happening the first time a script is
      seen, and the SHA1 computation is anyway not a very slow process against
      the typical Redis script and compared to the actua Lua byte compiling of
      the body.
      
      Note that the function used to assert() if a duplicated script was
      loaded, however actually now two times over three, we want the function
      to handle duplicated scripts just fine: this happens in SCRIPT LOAD and
      in RDB AUX "lua" loading. Moreover the assert was not defending against
      some obvious failure mode, so now the function always tests against
      already defined functions at start.
      60d26acf
  10. 01 Dec, 2017 5 commits
    • antirez's avatar
      Fix loading of RDB files lua AUX fields when the script is defined. · 65a9740f
      antirez authored
      In the case of slaves loading the RDB from master, or in other similar
      cases, the script is already defined, and the function registering the
      script should not fail in the assert() call.
      65a9740f
    • antirez's avatar
      Streams: delta encode IDs based on key. Add count + deleted fields. · f24d3a7d
      antirez authored
      We used to have the master ID stored at the start of the listpack,
      however using the key directly makes more sense in order to create a
      space efficient representation: anyway the key at the radix tree is very
      unlikely to change because of how the stream is implemented. Moreover on
      nodes merging, to rewrite the merged listpacks is anyway the most
      sensible operation, and we can use the iterator and the append-to-stream
      function in order to avoid re-implementing the code needed for merging.
      
      This commit also adds two items at the start of the listpack: the
      number of valid items inside the listpack, and the number of items
      marked as deleted. This means that there is no need to scan a listpack
      in order to understand if it's a good candidate for garbage collection,
      if the ration between valid/deleted items triggers the GC.
      f24d3a7d
    • antirez's avatar
      Streams: Save stream->length in RDB. · 98d184db
      antirez authored
      98d184db
    • antirez's avatar
      Streams: RDB loading. RDB saving modified. · edd70c19
      antirez authored
      After a few attempts it looked quite saner to just add the last item ID
      at the end of the serialized listpacks, instead of scanning the last
      listpack loaded from head to tail just to fetch it. It's a disk space VS
      CPU-and-simplicity tradeoff basically.
      edd70c19
    • antirez's avatar
      Streams: RDB saving. · 485014cc
      antirez authored
      485014cc
  11. 30 Nov, 2017 2 commits
    • antirez's avatar
      PSYNC2: just store script bodies into RDB. · 452ad2e9
      antirez authored
      Related to #4483. As suggested by @soloestoy, we can retrieve the SHA1
      from the body. Given that in the new implementation using AUX fields we
      ended copying around a lot to create new objects and strings, extremize
      such concept and trade CPU for space inside the RDB file.
      452ad2e9
    • antirez's avatar
      PSYNC2: Save Lua scripts state into RDB file. · f11a7585
      antirez authored
      This is currently needed in order to fix #4483, but this can be
      useful in other contexts, so maybe later we may want to remove the
      conditionals and always save/load scripts.
      
      Note that we are using the "lua" AUX field here, in order to guarantee
      backward compatibility of the RDB file. The unknown AUX fields must be
      discarded by past versions of Redis.
      f11a7585
  12. 24 Nov, 2017 1 commit
  13. 22 Nov, 2017 2 commits
  14. 06 Nov, 2017 1 commit
    • antirez's avatar
      Fix saving of zero-length lists. · a1944c3e
      antirez authored
      Normally in modern Redis you can't create zero-len lists, however it's
      possible to load them from old RDB files generated, for instance, using
      Redis 2.8 (see issue #4409). The "Right Thing" would be not loading such
      lists at all, but this requires to hook in rdb.c random places in a not
      great way, for a problem that is at this point, at best, minor.
      
      Here in this commit instead I just fix the fact that zero length lists,
      materialized as quicklists with the first node set to NULL, were
      iterated in the wrong way while they are saved, leading to a crash.
      
      The other parts of the list implementation are apparently able to deal
      with empty lists correctly, even if they are no longer a thing.
      a1944c3e
  15. 02 Nov, 2017 1 commit
  16. 01 Nov, 2017 1 commit
  17. 20 Sep, 2017 2 commits
    • antirez's avatar
      PSYNC2: More refinements related to #4316. · bb3b5ddd
      antirez authored
      bb3b5ddd
    • zhaozhao.zz's avatar
      PSYNC2: make persisiting replication info more solid · b541ccef
      zhaozhao.zz authored
      This commit is a reinforcement of commit c1c99e9f.
      
      1. Replication information can be stored when the RDB file is
      generated by a mater using server.slaveseldb when server.repl_backlog
      is not NULL, or set repl_stream_db be -1. That's safe, because
      NULL server.repl_backlog will trigger full synchronization,
      then master will send SELECT command to replicaiton stream.
      2. Only do rdbSave* when rsiptr is not NULL,
      if we do rdbSave* without rdbSaveInfo, slave will miss repl-stream-db.
      3. Save the replication informations also in the case of
      SAVE command, FLUSHALL command and DEBUG reload.
      b541ccef
  18. 19 Sep, 2017 1 commit
    • antirez's avatar
      PSYNC2: Fix the way replication info is saved/loaded from RDB. · c1c99e9f
      antirez authored
      This commit attempts to fix a number of bugs reported in #4316.
      They are related to the way replication info like replication ID,
      offsets, and currently selected DB in the master client, are stored
      and loaded by Redis. In order to avoid inconsistencies the changes in
      this commit try to enforce that:
      
      1. Replication information are only stored when the RDB file is
      generated by a slave that has a valid 'master' client, so that we can
      always extract the currently selected DB.
      2. When replication informations are persisted in the RDB file, all the
      info for a successful PSYNC or nothing is persisted.
      3. The RDB replication informations are only loaded if the instance is
      configured as a slave, otherwise a master can start with IDs that relate
      to a different history of the data set, and stil retain such IDs in the
      future while receiving unrelated writes.
      c1c99e9f
  19. 10 Jul, 2017 1 commit
  20. 06 Jul, 2017 1 commit
  21. 27 Jun, 2017 1 commit
    • antirez's avatar
      RDB modules values serialization format version 2. · 365dd037
      antirez authored
      The original RDB serialization format was not parsable without the
      module loaded, becuase the structure was managed only by the module
      itself. Moreover RDB is a streaming protocol in the sense that it is
      both produce di an append-only fashion, and is also sometimes directly
      sent to the socket (in the case of diskless replication).
      
      The fact that modules values cannot be parsed without the relevant
      module loaded is a problem in many ways: RDB checking tools must have
      loaded modules even for doing things not involving the value at all,
      like splitting an RDB into N RDBs by key or alike, or just checking the
      RDB for sanity.
      
      In theory module values could be just a blob of data with a prefixed
      length in order for us to be able to skip it. However prefixing the values
      with a length would mean one of the following:
      
      1. To be able to write some data at a previous offset. This breaks
      stremaing.
      2. To bufferize values before outputting them. This breaks performances.
      3. To have some chunked RDB output format. This breaks simplicity.
      
      Moreover, the above solution, still makes module values a totally opaque
      matter, with the fowllowing problems:
      
      1. The RDB check tool can just skip the value without being able to at
      least check the general structure. For datasets composed mostly of
      modules values this means to just check the outer level of the RDB not
      actually doing any checko on most of the data itself.
      2. It is not possible to do any recovering or processing of data for which a
      module no longer exists in the future, or is unknown.
      
      So this commit implements a different solution. The modules RDB
      serialization API is composed if well defined calls to store integers,
      floats, doubles or strings. After this commit, the parts generated by
      the module API have a one-byte prefix for each of the above emitted
      parts, and there is a final EOF byte as well. So even if we don't know
      exactly how to interpret a module value, we can always parse it at an
      high level, check the overall structure, understand the types used to
      store the information, and easily skip the whole value.
      
      The change is backward compatible: older RDB files can be still loaded
      since the new encoding has a new RDB type: MODULE_2 (of value 7).
      The commit also implements the ability to check RDB files for sanity
      taking advantage of the new feature.
      365dd037
  22. 19 May, 2017 1 commit