1. 08 Jan, 2015 2 commits
    • antirez's avatar
      A few more AUX info fields added to RDB. · bfbfd0dd
      antirez authored
      bfbfd0dd
    • antirez's avatar
      RDB AUX fields support. · dee23184
      antirez authored
      This commit introduces a new RDB data type called 'aux'. It is used in
      order to insert inside an RDB file key-value pairs that may serve
      different needs, without breaking backward compatibility when new
      informations are embedded inside an RDB file. The contract between Redis
      versions is to ignore unknown aux fields when encountered.
      
      Aux fields can be used in order to:
      
      1. Augment the RDB file with info like version of Redis that created the
      RDB file, creation time, used memory while the RDB was created, and so
      forth.
      2. Add state about Redis inside the RDB file that we need to reload
      later: replication offset, previos master run ID, in order to improve
      failovers safety and allow partial resynchronization after a slave
      restart.
      3. Anything that we may want to add to RDB files without breaking the
      ability of past versions of Redis to load the file.
      dee23184
  2. 07 Jan, 2015 4 commits
  3. 24 Dec, 2014 1 commit
  4. 23 Dec, 2014 33 commits
    • Matt Stancliff's avatar
      0b37730e
    • Matt Stancliff's avatar
      Set optional 'static' for Quicklist+Redis · 8842de1a
      Matt Stancliff authored
      This also defines REDIS_STATIC='' for building everything
      inside src/ and everything inside deps/lua/.
      8842de1a
    • Matt Stancliff's avatar
      Add more quicklist info to DEBUG OBJECT · 32266be2
      Matt Stancliff authored
      Adds: ql_compressed (boolean, 1 if compression enabled for list, 0
      otherwise)
      Adds: ql_uncompressed_size (actual uncompressed size of all quicklistNodes)
      Adds: ql_ziplist_max (quicklist max ziplist fill factor)
      
      Compression ratio of the list is then ql_uncompressed_size / serializedlength
      
      We report ql_uncompressed_size for all quicklists because serializedlength
      is a _compressed_ representation anyway.
      
      Sample output from a large list:
      127.0.0.1:6379> llen abc
      (integer) 38370061
      127.0.0.1:6379> debug object abc
      Value at:0x7ff97b51d140 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718164 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:0 ql_uncompressed_size:1643187761
      (1.36s)
      
      The 1.36s result time is because rdbSavedObjectLen() is serializing the
      object, not because of any new stats reporting.
      
      If we run DEBUG OBJECT on a compressed list, DEBUG OBJECT takes almost *zero*
      time because rdbSavedObjectLen() reuses already-compressed ziplists:
      127.0.0.1:6379> debug object abc
      Value at:0x7fe5c5800040 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718109 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:1 ql_uncompressed_size:1643187761
      32266be2
    • Matt Stancliff's avatar
      Config: Add quicklist, remove old list options · 919e8c30
      Matt Stancliff authored
      This removes:
        - list-max-ziplist-entries
        - list-max-ziplist-value
      
      This adds:
        - list-max-ziplist-size
        - list-compress-depth
      
      Also updates config file with new sections and updates
      tests to use quicklist settings instead of old list settings.
      919e8c30
    • Matt Stancliff's avatar
      Add branch prediction hints to quicklist · b49709e6
      Matt Stancliff authored
      Actually makes a noticeable difference.
      
      Branch hints were selected based on profiler hotspots.
      b49709e6
    • Matt Stancliff's avatar
      Allow compression of interior quicklist nodes · cf5750f2
      Matt Stancliff authored
      Let user set how many nodes to *not* compress.
      
      We can specify a compression "depth" of how many nodes
      to leave uncompressed on each end of the quicklist.
      
      Depth 0 = disable compression.
      Depth 1 = only leave head/tail uncompressed.
        - (read as: "skip 1 node on each end of the list before compressing")
      Depth 2 = leave head, head->next, tail->prev, tail uncompressed.
        - ("skip 2 nodes on each end of the list before compressing")
      Depth 3 = Depth 2 + head->next->next + tail->prev->prev
        - ("skip 3 nodes...")
      etc.
      
      This also:
        - updates RDB storage to use native quicklist compression (if node is
          already compressed) instead of uncompressing, generating the RDB string,
          then re-compressing the quicklist node.
        - internalizes the "fill" parameter for the quicklist so we don't
          need to pass it to _every_ function.  Now it's just a property of
          the list.
        - allows a runtime-configurable compression option, so we can
          expose a compresion parameter in the configuration file if people
          want to trade slight request-per-second performance for up to 90%+
          memory savings in some situations.
        - updates the quicklist tests to do multiple passes: 200k+ tests now.
      cf5750f2
    • Matt Stancliff's avatar
      Add quicklist info to DEBUG OBJECT · 50fe4145
      Matt Stancliff authored
      Added field 'ql_nodes' and 'ql_avg_per_node'.
      
      ql_nodes is the number of quicklist nodes in the quicklist.
      ql_avg_node is the average fill level in each quicklist node. (LLEN / QL_NODES)
      
      Sample output:
      127.0.0.1:6379> DEBUG object b
      Value at:0x7fa42bf2fed0 refcount:1 encoding:quicklist serializedlength:18489 lru:8983768 lru_seconds_idle:3 ql_nodes:430 ql_avg_per_node:511.73
      127.0.0.1:6379> llen b
      (integer) 220044
      50fe4145
    • Matt Stancliff's avatar
      Remove malloc failure checks · 26fe3736
      Matt Stancliff authored
      We trust zmalloc to kill the whole process on memory failure
      26fe3736
    • Matt Stancliff's avatar
      Increase test size for migrating large values · c865809c
      Matt Stancliff authored
      Previously, the old test ran 5,000 loops and used about 500k.
      
      With quicklist, storing those same 5,000 loops takes up 24k, so the
      "large value check" failed!
      
      This increases the test to 20,000 loops which makes the object dump 96k.
      c865809c
    • Matt Stancliff's avatar
      Convert quicklist RDB to store ziplist nodes · 8919e153
      Matt Stancliff authored
      Turns out it's a huge improvement during save/reload/migrate/restore
      because, with compression enabled, we're compressing 4k or 8k
      chunks of data consisting of multiple elements in one ziplist
      instead of compressing series of smaller individual elements.
      8919e153
    • Matt Stancliff's avatar
      Convert RDB ziplist loading to sdsnative() · 73582497
      Matt Stancliff authored
      This saves us an unnecessary zmalloc, memcpy, and two frees.
      73582497
    • Matt Stancliff's avatar
      Add sdsnative() · 23a93d5c
      Matt Stancliff authored
      Use the existing memory space for an SDS to convert it to a regular
      character buffer so we don't need to allocate duplicate space just
      to extract a usable buffer for native operations.
      23a93d5c
    • Matt Stancliff's avatar
      Add adaptive quicklist fill factor · cb8032d6
      Matt Stancliff authored
      Fill factor now has two options:
        - negative (1-5) for size-based ziplist filling
        - positive for length-based ziplist filling with implicit size cap.
      
      Negative offsets define ziplist size limits of:
        -1: 4k
        -2: 8k
        -3: 16k
        -4: 32k
        -5: 64k
      
      Positive offsets now automatically limit their max size to 8k.  Any
      elements larger than 8k will be in individual nodes.
      
      Positive ziplist fill factors will keep adding elements
      to a ziplist until one of:
        - ziplist has FILL number of elements
          - or -
        - ziplist grows above our ziplist max size (currently 8k)
      
      When using positive fill factors, if you insert a large
      element (over 8k), that element will automatically allocate
      an individual quicklist node with one element and no other elements will be
      in the same ziplist inside that quicklist node.
      
      When using negative fill factors, elements up to the size
      limit can be added to one quicklist node.  If an element
      is added larger than the max ziplist size, that element
      will be allocated an individual ziplist in a new quicklist node.
      
      Tests also updated to start testing at fill factor -5.
      cb8032d6
    • Matt Stancliff's avatar
      redis-benchmark: Add RPUSH and RPOP tests · 2cf51b46
      Matt Stancliff authored
      2cf51b46
    • Matt Stancliff's avatar
      Free ziplist test lists during tests · 36f7ca08
      Matt Stancliff authored
      Freeing our test lists helps keep valgrind output clean
      36f7ca08
    • Matt Stancliff's avatar
      Add ziplistMerge() · b39bfb32
      Matt Stancliff authored
      This started out as #2158 by sunheehnus, but I kept rewriting it
      until I could understand things more easily and get a few more
      correctness guarantees out of the readability flow.
      
      The original commit created and returned a new ziplist with the contents of
      both input ziplists, but I prefer to grow one of the input ziplists
      and destroy the other one.
      
      So, instead of malloc+copy as in #2158, the merge now reallocs one of
      the existing ziplists and copies the other ziplist into the new space.
      
      Also added merge test cases to ziplistTest()
      b39bfb32
    • Matt Stancliff's avatar
      Add quicklist implementation · ff9e5b21
      Matt Stancliff authored
      This replaces individual ziplist vs. linkedlist representations
      for Redis list operations.
      
      Big thanks for all the reviews and feedback from everybody in
      https://github.com/antirez/redis/pull/2143
      ff9e5b21
    • Matt Stancliff's avatar
      Fix three simple clang analyzer warnings · d956d809
      Matt Stancliff authored
      d956d809
    • Matt Stancliff's avatar
      Cleanup ziplist valgrind warnings · 9b786b12
      Matt Stancliff authored
      Valgrind can't detect 'memset' initializes things, so let's
      statically initialize them to remove some unnecessary warnings.
      9b786b12
    • Matt Stancliff's avatar
      Fix ziplist test for pop() · 1dfcd75a
      Matt Stancliff authored
      The previous test wasn't returning the new ziplist, so the test
      was invalid.  Now the test works properly.
      
      These problems were simultaenously discovered in #2154 and that
      PR also had an additional fix we included here.
      1dfcd75a
    • Matt Stancliff's avatar
      Fix ziplistDeleteRange index parameter · 53b1ee34
      Matt Stancliff authored
      It's valid to delete from negative offsets, so we *don't*
      want unsigned arguments here.
      53b1ee34
    • Matt Stancliff's avatar
      Fix how zipEntry returns values · fae53dea
      Matt Stancliff authored
      zipEntry was returning a struct, but that caused some
      problems with tests under 32 bit builds.
      
      The tests run better if we operate on structs allocated in the
      caller without worrying about copying on return.
      fae53dea
    • Matt Stancliff's avatar
      Allow forcing non-jemalloc build · d01d4ddc
      Matt Stancliff authored
      d01d4ddc
    • Matt Stancliff's avatar
      Add simple ll2string() tests · 9b343678
      Matt Stancliff authored
      9b343678
    • Matt Stancliff's avatar
      Allow all code tests to run using Redis args · 8febcffd
      Matt Stancliff authored
      Previously, many files had individual main() functions for testing,
      but each required being compiled with their own testing flags.
      That gets difficult when you have 8 different flags you need
      to set just to run all tests (plus, some test files required
      other files to be compiled aaginst them, and it seems some didn't
      build at all without including the rest of Redis).
      
      Now all individual test main() funcions are renamed to a test
      function for the file itself and one global REDIS_TEST define enables
      testing across the entire codebase.
      
      Tests can now be run with:
        - `./redis-server test <test>`
      
        e.g. ./redis-server test ziplist
      
      If REDIS_TEST is not defined, then no tests get included and no
      tests are included in the final redis-server binary.
      8febcffd
    • Matt Stancliff's avatar
      Remove ziplist compiler warnings · 8380655e
      Matt Stancliff authored
      Only happen when compiled with the test define.
      8380655e
    • Matt Stancliff's avatar
      Add DEBUG JEMALLOC INFO · 27937c28
      Matt Stancliff authored
      Uses jemalloc function malloc_stats_print() to return
      stats about what jemalloc has allocated internally.
      27937c28
    • Matt Stancliff's avatar
      Add addReplyBulkSds() function · ad41a7c4
      Matt Stancliff authored
      Refactor a common pattern into one function so we don't
      end up with copy/paste programming.
      ad41a7c4
    • antirez's avatar
    • antirez's avatar
      INFO loading stats: three fixes. · 840435ad
      antirez authored
      1. Server unxtime may remain not updated while loading AOF, so ETA is
      not updated correctly.
      
      2. Number of processed byte was not initialized.
      
      3. Possible division by zero condition (likely cause of issue #1932).
      840435ad
    • Salvatore Sanfilippo's avatar
      Merge pull request #2227 from mattsta/fix/trib/assignment/master-replica · f7bc1fce
      Salvatore Sanfilippo authored
      Improve redis-trib replica assignment
      f7bc1fce
    • Salvatore Sanfilippo's avatar
      Merge pull request #2234 from mattsta/feature/sentinel-info-cache/age · 3a1ee895
      Salvatore Sanfilippo authored
      Add 'age' value to SENTINEL INFO-CACHE
      3a1ee895
    • Matt Stancliff's avatar
      Add 'age' value to SENTINEL INFO-CACHE · 32bba43a
      Matt Stancliff authored
      32bba43a