1. 08 Jan, 2015 5 commits
  2. 02 Jan, 2015 17 commits
    • Matt Stancliff's avatar
      5870e224
    • Matt Stancliff's avatar
      Set optional 'static' for Quicklist+Redis · 25e12d10
      Matt Stancliff authored
      This also defines REDIS_STATIC='' for building everything
      inside src/ and everything inside deps/lua/.
      25e12d10
    • Matt Stancliff's avatar
      Add more quicklist info to DEBUG OBJECT · 9e11d079
      Matt Stancliff authored
      Adds: ql_compressed (boolean, 1 if compression enabled for list, 0
      otherwise)
      Adds: ql_uncompressed_size (actual uncompressed size of all quicklistNodes)
      Adds: ql_ziplist_max (quicklist max ziplist fill factor)
      
      Compression ratio of the list is then ql_uncompressed_size / serializedlength
      
      We report ql_uncompressed_size for all quicklists because serializedlength
      is a _compressed_ representation anyway.
      
      Sample output from a large list:
      127.0.0.1:6379> llen abc
      (integer) 38370061
      127.0.0.1:6379> debug object abc
      Value at:0x7ff97b51d140 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718164 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:0 ql_uncompressed_size:1643187761
      (1.36s)
      
      The 1.36s result time is because rdbSavedObjectLen() is serializing the
      object, not because of any new stats reporting.
      
      If we run DEBUG OBJECT on a compressed list, DEBUG OBJECT takes almost *zero*
      time because rdbSavedObjectLen() reuses already-compressed ziplists:
      127.0.0.1:6379> debug object abc
      Value at:0x7fe5c5800040 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718109 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:1 ql_uncompressed_size:1643187761
      9e11d079
    • Matt Stancliff's avatar
      Config: Add quicklist, remove old list options · 02bb515a
      Matt Stancliff authored
      This removes:
        - list-max-ziplist-entries
        - list-max-ziplist-value
      
      This adds:
        - list-max-ziplist-size
        - list-compress-depth
      
      Also updates config file with new sections and updates
      tests to use quicklist settings instead of old list settings.
      02bb515a
    • Matt Stancliff's avatar
      Add branch prediction hints to quicklist · bbbbfb14
      Matt Stancliff authored
      Actually makes a noticeable difference.
      
      Branch hints were selected based on profiler hotspots.
      bbbbfb14
    • Matt Stancliff's avatar
      Cleanup quicklist style · 5f506b6d
      Matt Stancliff authored
      Small fixes due to a new version of clang-format (it's less
      crazy than the older version).
      5f506b6d
    • Matt Stancliff's avatar
      Allow compression of interior quicklist nodes · abdd1414
      Matt Stancliff authored
      Let user set how many nodes to *not* compress.
      
      We can specify a compression "depth" of how many nodes
      to leave uncompressed on each end of the quicklist.
      
      Depth 0 = disable compression.
      Depth 1 = only leave head/tail uncompressed.
        - (read as: "skip 1 node on each end of the list before compressing")
      Depth 2 = leave head, head->next, tail->prev, tail uncompressed.
        - ("skip 2 nodes on each end of the list before compressing")
      Depth 3 = Depth 2 + head->next->next + tail->prev->prev
        - ("skip 3 nodes...")
      etc.
      
      This also:
        - updates RDB storage to use native quicklist compression (if node is
          already compressed) instead of uncompressing, generating the RDB string,
          then re-compressing the quicklist node.
        - internalizes the "fill" parameter for the quicklist so we don't
          need to pass it to _every_ function.  Now it's just a property of
          the list.
        - allows a runtime-configurable compression option, so we can
          expose a compresion parameter in the configuration file if people
          want to trade slight request-per-second performance for up to 90%+
          memory savings in some situations.
        - updates the quicklist tests to do multiple passes: 200k+ tests now.
      abdd1414
    • Matt Stancliff's avatar
      Add quicklist info to DEBUG OBJECT · 5127e399
      Matt Stancliff authored
      Added field 'ql_nodes' and 'ql_avg_per_node'.
      
      ql_nodes is the number of quicklist nodes in the quicklist.
      ql_avg_node is the average fill level in each quicklist node. (LLEN / QL_NODES)
      
      Sample output:
      127.0.0.1:6379> DEBUG object b
      Value at:0x7fa42bf2fed0 refcount:1 encoding:quicklist serializedlength:18489 lru:8983768 lru_seconds_idle:3 ql_nodes:430 ql_avg_per_node:511.73
      127.0.0.1:6379> llen b
      (integer) 220044
      5127e399
    • Matt Stancliff's avatar
      Remove malloc failure checks · 8d702189
      Matt Stancliff authored
      We trust zmalloc to kill the whole process on memory failure
      8d702189
    • Matt Stancliff's avatar
      Convert quicklist RDB to store ziplist nodes · 101b3a6e
      Matt Stancliff authored
      Turns out it's a huge improvement during save/reload/migrate/restore
      because, with compression enabled, we're compressing 4k or 8k
      chunks of data consisting of multiple elements in one ziplist
      instead of compressing series of smaller individual elements.
      101b3a6e
    • Matt Stancliff's avatar
      Convert RDB ziplist loading to sdsnative() · 127c15e2
      Matt Stancliff authored
      This saves us an unnecessary zmalloc, memcpy, and two frees.
      127c15e2
    • Matt Stancliff's avatar
      Add sdsnative() · e1619772
      Matt Stancliff authored
      Use the existing memory space for an SDS to convert it to a regular
      character buffer so we don't need to allocate duplicate space just
      to extract a usable buffer for native operations.
      e1619772
    • Matt Stancliff's avatar
      Add adaptive quicklist fill factor · c6bf20c2
      Matt Stancliff authored
      Fill factor now has two options:
        - negative (1-5) for size-based ziplist filling
        - positive for length-based ziplist filling with implicit size cap.
      
      Negative offsets define ziplist size limits of:
        -1: 4k
        -2: 8k
        -3: 16k
        -4: 32k
        -5: 64k
      
      Positive offsets now automatically limit their max size to 8k.  Any
      elements larger than 8k will be in individual nodes.
      
      Positive ziplist fill factors will keep adding elements
      to a ziplist until one of:
        - ziplist has FILL number of elements
          - or -
        - ziplist grows above our ziplist max size (currently 8k)
      
      When using positive fill factors, if you insert a large
      element (over 8k), that element will automatically allocate
      an individual quicklist node with one element and no other elements will be
      in the same ziplist inside that quicklist node.
      
      When using negative fill factors, elements up to the size
      limit can be added to one quicklist node.  If an element
      is added larger than the max ziplist size, that element
      will be allocated an individual ziplist in a new quicklist node.
      
      Tests also updated to start testing at fill factor -5.
      c6bf20c2
    • Matt Stancliff's avatar
      redis-benchmark: Add RPUSH and RPOP tests · 60a9418e
      Matt Stancliff authored
      60a9418e
    • Matt Stancliff's avatar
      Free ziplist test lists during tests · 0f15eb18
      Matt Stancliff authored
      Freeing our test lists helps keep valgrind output clean
      0f15eb18
    • Matt Stancliff's avatar
      Add ziplistMerge() · 9d2dc024
      Matt Stancliff authored
      This started out as #2158 by sunheehnus, but I kept rewriting it
      until I could understand things more easily and get a few more
      correctness guarantees out of the readability flow.
      
      The original commit created and returned a new ziplist with the contents of
      both input ziplists, but I prefer to grow one of the input ziplists
      and destroy the other one.
      
      So, instead of malloc+copy as in #2158, the merge now reallocs one of
      the existing ziplists and copies the other ziplist into the new space.
      
      Also added merge test cases to ziplistTest()
      9d2dc024
    • Matt Stancliff's avatar
      Add quicklist implementation · 5e362b84
      Matt Stancliff authored
      This replaces individual ziplist vs. linkedlist representations
      for Redis list operations.
      
      Big thanks for all the reviews and feedback from everybody in
      https://github.com/antirez/redis/pull/2143
      5e362b84
  3. 23 Dec, 2014 13 commits
    • Matt Stancliff's avatar
      Fix three simple clang analyzer warnings · d956d809
      Matt Stancliff authored
      d956d809
    • Matt Stancliff's avatar
      Cleanup ziplist valgrind warnings · 9b786b12
      Matt Stancliff authored
      Valgrind can't detect 'memset' initializes things, so let's
      statically initialize them to remove some unnecessary warnings.
      9b786b12
    • Matt Stancliff's avatar
      Fix ziplist test for pop() · 1dfcd75a
      Matt Stancliff authored
      The previous test wasn't returning the new ziplist, so the test
      was invalid.  Now the test works properly.
      
      These problems were simultaenously discovered in #2154 and that
      PR also had an additional fix we included here.
      1dfcd75a
    • Matt Stancliff's avatar
      Fix ziplistDeleteRange index parameter · 53b1ee34
      Matt Stancliff authored
      It's valid to delete from negative offsets, so we *don't*
      want unsigned arguments here.
      53b1ee34
    • Matt Stancliff's avatar
      Fix how zipEntry returns values · fae53dea
      Matt Stancliff authored
      zipEntry was returning a struct, but that caused some
      problems with tests under 32 bit builds.
      
      The tests run better if we operate on structs allocated in the
      caller without worrying about copying on return.
      fae53dea
    • Matt Stancliff's avatar
      Allow forcing non-jemalloc build · d01d4ddc
      Matt Stancliff authored
      d01d4ddc
    • Matt Stancliff's avatar
      Add simple ll2string() tests · 9b343678
      Matt Stancliff authored
      9b343678
    • Matt Stancliff's avatar
      Allow all code tests to run using Redis args · 8febcffd
      Matt Stancliff authored
      Previously, many files had individual main() functions for testing,
      but each required being compiled with their own testing flags.
      That gets difficult when you have 8 different flags you need
      to set just to run all tests (plus, some test files required
      other files to be compiled aaginst them, and it seems some didn't
      build at all without including the rest of Redis).
      
      Now all individual test main() funcions are renamed to a test
      function for the file itself and one global REDIS_TEST define enables
      testing across the entire codebase.
      
      Tests can now be run with:
        - `./redis-server test <test>`
      
        e.g. ./redis-server test ziplist
      
      If REDIS_TEST is not defined, then no tests get included and no
      tests are included in the final redis-server binary.
      8febcffd
    • Matt Stancliff's avatar
      Remove ziplist compiler warnings · 8380655e
      Matt Stancliff authored
      Only happen when compiled with the test define.
      8380655e
    • Matt Stancliff's avatar
      Add DEBUG JEMALLOC INFO · 27937c28
      Matt Stancliff authored
      Uses jemalloc function malloc_stats_print() to return
      stats about what jemalloc has allocated internally.
      27937c28
    • Matt Stancliff's avatar
      Add addReplyBulkSds() function · ad41a7c4
      Matt Stancliff authored
      Refactor a common pattern into one function so we don't
      end up with copy/paste programming.
      ad41a7c4
    • antirez's avatar
      INFO loading stats: three fixes. · 840435ad
      antirez authored
      1. Server unxtime may remain not updated while loading AOF, so ETA is
      not updated correctly.
      
      2. Number of processed byte was not initialized.
      
      3. Possible division by zero condition (likely cause of issue #1932).
      840435ad
    • Matt Stancliff's avatar
      Add 'age' value to SENTINEL INFO-CACHE · 32bba43a
      Matt Stancliff authored
      32bba43a
  4. 21 Dec, 2014 2 commits
  5. 20 Dec, 2014 1 commit
  6. 19 Dec, 2014 1 commit
  7. 18 Dec, 2014 1 commit
    • Alon Diamant's avatar
      Fix: case when SPOP with count>MAXINT, setTypeRandomElements() will get... · 3c8a7558
      Alon Diamant authored
      Fix: case when SPOP with count>MAXINT, setTypeRandomElements() will get negative count argument due to signed/unsigned mismatch.
      
      setTypeRandomElements() now returns unsigned long, and also uses unsigned long for anything related to count of members.
      spopWithCountCommand() now uses unsigned long elements_returned instead of int, for values returned from setTypeRandomElements()
      3c8a7558