1. 18 Nov, 2021 1 commit
    • Binbin's avatar
      Fixes ZPOPMIN/ZPOPMAX wrong replies when count is 0 with non-zset (#9711) · 91e77a0c
      Binbin authored
      Moves ZPOP ... 0 fast exit path after type check to reply with
      WRONGTYPE. In the past it will return an empty array.
      
      Also now count is not allowed to be negative.
      
      see #9680
      
      before:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (empty array)
      127.0.0.1:6379> zpopmin zset -1
      (empty array)
      ```
      
      after:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (error) WRONGTYPE Operation against a key holding the wrong kind of value
      127.0.0.1:6379> zpopmin zset -1
      (error) ERR value is out of range, must be positive
      ```
      91e77a0c
  2. 03 Nov, 2021 1 commit
    • perryitay's avatar
      fix: lookupKey on SETNX and SETXX only once (#9640) · 77d3c6bf
      perryitay authored
      When using SETNX and SETXX we could end up doing key lookup twice.
      This presents a small inefficiency price.
      Also once we have statistics of write hit and miss they'll be wrong (recording the same key hit twice) 
      77d3c6bf
  3. 31 Oct, 2021 1 commit
    • Binbin's avatar
      Fix multiple COUNT in LMPOP/BLMPOP/ZMPOP/BZMPOP (#9701) · 03357883
      Binbin authored
      The previous code did not check whether COUNT is set.
      So we can use `lmpop 2 key1 key2 left count 1 count 2`.
      
      This situation can occur in LMPOP/BLMPOP/ZMPOP/BZMPOP commands.
      LMPOP/BLMPOP introduced in #9373, ZMPOP/BZMPOP introduced in #9484.
      03357883
  4. 04 Oct, 2021 1 commit
    • Oran Agra's avatar
      Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) (#9589) · c5e6a620
      Oran Agra authored
      
      
      - fix possible heap corruption in ziplist and listpack resulting by trying to
        allocate more than the maximum size of 4GB.
      - prevent ziplist (hash and zset) from reaching size of above 1GB, will be
        converted to HT encoding, that's not a useful size.
      - prevent listpack (stream) from reaching size of above 1GB.
      - XADD will start a new listpack if the new record may cause the previous
        listpack to grow over 1GB.
      - XADD will respond with an error if a single stream record is over 1GB
      - List type (ziplist in quicklist) was truncating strings that were over 4GB,
        now it'll respond with an error.
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      c5e6a620
  5. 03 Oct, 2021 1 commit
    • Binbin's avatar
      Cleanup typos, incorrect comments, and fixed small memory leak in redis-cli (#9153) · dd3ac97f
      Binbin authored
      1. Remove forward declarations from header files to functions that do not exist:
      hmsetCommand and rdbSaveTime.
      2. Minor phrasing fixes in #9519
      3. Add missing sdsfree(title) and fix typo in redis-benchmark.
      4. Modify some error comments in some zset commands.
      5. Fix copy-paste bug comment in syncWithMaster about `ip-address`.
      dd3ac97f
  6. 24 Sep, 2021 1 commit
    • sundb's avatar
      Use dictGetFairRandomKey() for HRANDFIELD,SRANDMEMBER,ZRANDMEMBER (#9538) · 9967a53f
      sundb authored
      In the `HRANDFIELD`, `SRANDMEMBER` and `ZRANDMEMBER` commands,
      There are some strategies that could in some rare cases return an unfair random.
      these cases are where s small dict happens be be hashed unevenly.
      
      Specifically when `count*ZRANDMEMBER_SUB_STRATEGY_MUL > size`,
      using `dictGetRandomKey` to randomize from a dict will result in an unfair random result.
      9967a53f
  7. 23 Sep, 2021 1 commit
    • Binbin's avatar
      Add ZMPOP/BZMPOP commands. (#9484) · 14d6abd8
      Binbin authored
      This is similar to the recent addition of LMPOP/BLMPOP (#9373), but zset.
      
      Syntax for the new ZMPOP command:
      `ZMPOP numkeys [<key> ...] MIN|MAX [COUNT count]`
      
      Syntax for the new BZMPOP command:
      `BZMPOP timeout numkeys [<key> ...] MIN|MAX [COUNT count]`
      
      Some background:
      - ZPOPMIN/ZPOPMAX take only one key, and can return multiple elements.
      - BZPOPMIN/BZPOPMAX take multiple keys, but return only one element from just one key.
      - ZMPOP/BZMPOP can take multiple keys, and can return multiple elements from just one key.
      
      Note that ZMPOP/BZMPOP can take multiple keys, it eventually operates on just on key.
      And it will propagate as ZPOPMIN or ZPOPMAX with the COUNT option.
      
      As new commands, if we can not pop any elements, the response like:
      - ZMPOP: Return a NIL in both RESP2 and RESP3, unlike ZPOPMIN/ZPOPMAX return emptyarray.
      - BZMPOP: Return a NIL in both RESP2 and RESP3 when timeout is reached, like BZPOPMIN/BZPOPMAX.
      
      For the normal response is nested arrays in RESP2 and RESP3:
      ```
      ZMPOP/BZMPOP
      1) keyname
      2) 1) 1) member1
            2) score1
         2) 1) member2
            2) score2
      
      In RESP2:
      1) "myzset"
      2) 1) 1) "three"
            2) "3"
         2) 1) "two"
            2) "2"
      
      In RESP3:
      1) "myzset"
      2) 1) 1) "three"
            2) (double) 3
         2) 1) "two"
            2) (double) 2
      ```
      14d6abd8
  8. 16 Sep, 2021 1 commit
    • Binbin's avatar
      Adds limit to SINTERCARD/ZINTERCARD. (#9425) · f898a9e9
      Binbin authored
      Implements the [LIMIT limit] variant of SINTERCARD/ZINTERCARD.
      Now with the LIMIT, we can stop the searching when cardinality
      reaching the limit, and return the cardinality ASAP.
      
      Note that in SINTERCARD, the old synatx was: `SINTERCARD key [key ...]`
      In order to add a optional parameter, we must break the old synatx.
      So the new syntax of SINTERCARD will be consistent with ZINTERCARD.
      New syntax: `SINTERCARD numkeys key [key ...] [LIMIT limit]`.
      
      Note that this means that SINTERCARD has a different syntax than
      SINTER and SINTERSTORE (taking numkeys argument)
      
      As for ZINTERCARD, we can easily add a optional parameter to it.
      New syntax: `ZINTERCARD numkeys key [key ...] [LIMIT limit]`
      f898a9e9
  9. 09 Sep, 2021 2 commits
    • sundb's avatar
      Replace all usage of ziplist with listpack for t_zset (#9366) · 3ca6972e
      sundb authored
      Part two of implementing #8702 (zset), after #8887.
      
      ## Description of the feature
      Replaced all uses of ziplist with listpack in t_zset, and optimized some of the code to optimize performance.
      
      ## Rdb format changes
      New `RDB_TYPE_ZSET_LISTPACK` rdb type.
      
      ## Rdb loading improvements:
      1) Pre-expansion of dict for validation of duplicate data for listpack and ziplist.
      2) Simplifying the release of empty key objects when RDB loading.
      3) Unify ziplist and listpack data verify methods for zset and hash, and move code to rdb.c.
      
      ## Interface changes
      1) New `zset-max-listpack-entries` config is an alias for `zset-max-ziplist-entries` (same with `zset-max-listpack-value`).
      2) OBJECT ENCODING will return listpack instead of ziplist.
      
      ## Listpack improvements:
      1) Add `lpDeleteRange` and `lpDeleteRangeWithEntry` functions to delete a range of entries from listpack.
      2) Improve the performance of `lpCompare`, converting from string to integer is faster than converting from integer to string.
      3) Replace `snprintf` with `ll2string` to improve performance in converting numbers to strings in `lpGet()`.
      
      ## Zset improvements:
      1) Improve the performance of `zzlFind` method, use `lpFind` instead of `lpCompare` in a loop.
      2) Use `lpDeleteRangeWithEntry` instead of `lpDelete` twice to delete a element of zset.
      
      ## Tests
      1) Add some unittests for `lpDeleteRange` and `lpDeleteRangeWithEntry` function.
      2) Add zset RDB loading test.
      3) Add benchmark test for `lpCompare` and `ziplsitCompare`.
      4) Add empty listpack zset corrupt dump test.
      3ca6972e
    • Binbin's avatar
      Add LMPOP/BLMPOP commands. (#9373) · c50af0ae
      Binbin authored
      We want to add COUNT option for BLPOP.
      But we can't do it without breaking compatibility due to the command arguments syntax.
      So this commit introduce two new commands.
      
      Syntax for the new LMPOP command:
      `LMPOP numkeys [<key> ...] LEFT|RIGHT [COUNT count]`
      
      Syntax for the new BLMPOP command:
      `BLMPOP timeout numkeys [<key> ...] LEFT|RIGHT [COUNT count]`
      
      Some background:
      - LPOP takes one key, and can return multiple elements.
      - BLPOP takes multiple keys, but returns one element from just one key.
      - LMPOP can take multiple keys and return multiple elements from just one key.
      
      Note that LMPOP/BLMPOP  can take multiple keys, it eventually operates on just one key.
      And it will propagate as LPOP or RPOP with the COUNT option.
      
      As a new command, it still return NIL if we can't pop any elements.
      For the normal response is nested arrays in RESP2 and RESP3, like:
      ```
      LMPOP/BLMPOP 
      1) keyname
      2) 1) element1
         2) element2
      ```
      I.e. unlike BLPOP that returns a key name and one element so it uses a flat array,
      and LPOP that returns multiple elements with no key name, and again uses a flat array,
      this one has to return a nested array, and it does for for both RESP2 and RESP3 (like SCAN does)
      
      Some discuss can see: #766 #8824
      c50af0ae
  10. 05 Aug, 2021 1 commit
    • yoav-steinberg's avatar
      dict struct memory optimizations (#9228) · 5e908a29
      yoav-steinberg authored
      Reduce dict struct memory overhead
      on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin.
      
      summary of changes:
      - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below).
      - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c)
      - Eliminate the `sizemask` field, can be calculated from size when needed.
      - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8.
      
      Interface change: pass dict pointer to dict type call back functions.
      This is instead of passing the removed privdata field. In the future if
      we'd like to have private data in the callbacks we can extract it from
      the dict type. We can extend dictType to include a custom dict struct
      allocator and use it to allocate more data at the end of the dict
      struct. This data can then be used to store private data later acccessed
      by the callbacks.
      5e908a29
  11. 03 Aug, 2021 1 commit
  12. 22 Jul, 2021 1 commit
    • Oran Agra's avatar
      fix zslGetRank bug in dead-code (#9246) · 9ca5e8c5
      Oran Agra authored
      
      
      This fixes an issue with zslGetRank which will happen only if the
      skiplist data stracture is added two entries with the same element name,
      this can't happen in redis zsets (we use dict), but in theory this is a
      bug in the underlaying skiplist code.
      
      Fixes #3081 and #4032
      Co-authored-by: default avatarminjian.cai <cmjgithub@163.com>
      9ca5e8c5
  13. 18 Jul, 2021 1 commit
  14. 05 Jul, 2021 1 commit
  15. 29 Jun, 2021 2 commits
  16. 22 Jun, 2021 1 commit
  17. 16 Jun, 2021 2 commits
    • Jason Elbaum's avatar
      Change return value type for ZPOPMAX/MIN in RESP3 (#8981) · 7f342020
      Jason Elbaum authored
      When using RESP3, ZPOPMAX/ZPOPMIN should return nested arrays for consistency
      with other commands (e.g. ZRANGE).
      
      We do that only when COUNT argument is present (similarly to how LPOP behaves).
      for reasoning see https://github.com/redis/redis/issues/8824#issuecomment-855427955
      
      This is a breaking change only when RESP3 is used, and COUNT argument is present!
      7f342020
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  18. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  19. 06 Jun, 2021 1 commit
  20. 15 Apr, 2021 1 commit
  21. 14 Apr, 2021 1 commit
  22. 01 Apr, 2021 2 commits
    • guybe7's avatar
      zsetAdd: Fix wrong reply in case of INCR and GT/LT (#8717) · 843f769b
      guybe7 authored
      If GT/LT fails the operation we need to reply with
      nill (like failure due to NX).
      
      Other changes:
      Add the missing $encoding suffix to many zset tests
      
      Note: there's a behavior change just in case of INCR + GT/LT that fails.
      The old code was replying with the wrong (rejected) score, and now it'll reply with nil.
      
      Note that that's anyway a corner case so this "behavior change" shouldn't have too much affect.
      Using GT/LT with INCR has a predictable result even before we run the command
      (INCR GT will only only / always fail if the increment is negative).
      843f769b
    • wuYin's avatar
      reuse existing range comparators in the zset (#8714) · e5d50b23
      wuYin authored
      There are 2 common range comparators for skiplist: zslValueGteMin and
      zslValueLteMax, but they're not being reused in zslDeleteRangeByScore
      
      This is a small change to make code cleaner.
      e5d50b23
  23. 10 Mar, 2021 1 commit
    • guybe7's avatar
      Cleanup ZADD_* flags (#8559) · 61a73de6
      guybe7 authored
      Have a clear separation between in and out flags
      
      Other changes:
      
      delete dead code in RM_ZsetIncrby: if zsetAdd returned error (happens only if
      the result of the operation is NAN or if score is NAN) we return immediately so
      there is no way that zsetAdd succeeded and returned NAN in the out-flags
      61a73de6
  24. 22 Feb, 2021 1 commit
    • Wen Hui's avatar
      SRANDMEMBER RESP3 return should be Array, not Set (#8504) · f5235b2d
      Wen Hui authored
      
      
      SRANDMEMBER with negative count (non unique) can return the same member
      multiple times, and the order of elements in the returned collection matters.
      For these reasons returning a RESP3 Set type is not valid for the negative
      count, but also not really valid for the positive (unique) variant either (the
      command returns an array of random picks, not a set)
      
      This PR also contains a minor optimization for SRANDMEMBER, HRANDFIELD,
      and ZRANDMEMBER, to avoid the temporary dict from being rehashed while it grows.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      f5235b2d
  25. 07 Feb, 2021 1 commit
    • Oran Agra's avatar
      Optimize HRANDFIELD and ZRANDMEMBER case 4 when ziplist encoded (#8444) · 62b1f320
      Oran Agra authored
      
      
      It is inefficient to repeatedly pick a single random element from a
      ziplist.
      For CASE4, which is when the user requested a low number of unique
      random picks from the collectoin, we used thta pattern.
      
      Now we use a different algorithm that picks unique elements from a
      ziplist, and guarentee no duplicate but doesn't provide random order
      (which is only needed in the non-unique random picks case)
      
      Unrelated changes:
      * change ziplist count and indexes variables to unsigned
      * solve compilation warnings about uninitialized vars in gcc 10.2
      Co-authored-by: default avatarxinluton <xinluton@qq.com>
      62b1f320
  26. 05 Feb, 2021 1 commit
    • sundb's avatar
      RAND* commands: fix risk of OOM panic in hash and zset, use fair random in... · 18ac4197
      sundb authored
      
      RAND* commands: fix risk of OOM panic in hash and zset, use fair random in hash, and add tests for even distribution to all (#8429)
      
      Changes to HRANDFIELD and ZRANDMEMBER:
      * Fix risk of OOM panic when client query a very big negative count (avoid allocating huge temporary buffer).
      * Fix uneven random distribution in HRANDFIELD with negative count (wasn't using dictGetFairRandomKey).
      * Add tests to check an even random distribution (HRANDFIELD, SRANDMEMBER, ZRANDMEMBER).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      18ac4197
  27. 29 Jan, 2021 1 commit
    • Yang Bodong's avatar
      Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297) · b9a0500f
      Yang Bodong authored
      
      
      New commands:
      `HRANDFIELD [<count> [WITHVALUES]]`
      `ZRANDMEMBER [<count> [WITHSCORES]]`
      Algorithms are similar to the one in SRANDMEMBER.
      
      Both return a simple bulk response when no arguments are given, and an array otherwise.
      In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
      note: in all 3 commands, the only option that also provides random order is the one with negative count.
      
      Changes to SRANDMEMBER
      * Optimization when count is 1, we can use the more efficient algorithm of non-unique random
      * optimization: work with sds strings rather than robj
      
      Other changes:
      * zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
        case the buffer is too small)
      * Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
        case 4) and by accident used a negative count
      Co-authored-by: default avatarxinluton <xinluton@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b9a0500f
  28. 26 Jan, 2021 2 commits
  29. 13 Jan, 2021 1 commit
    • sundb's avatar
      Fix use of lookupKeyRead and lookupKeyWrite in zrangeGenericCommand,... · 8a81ed1b
      sundb authored
      Fix use of  lookupKeyRead and lookupKeyWrite in zrangeGenericCommand, zunionInterDiffGenericCommand (#8316)
      
      * Change zunionInterDiffGenericCommand to use lookupKeyRead if dstkey is null
      * Change zrangeGenericCommand to use lookupKey Write if dstkey isn't null
      
      ZRANGESTORE and UNION, ZINTER, ZDIFF are all new commands (6.2 RC1 and RC2).
      In redis 6.0 the ZRANGE was using lookupKeyRead, and ZUNIONSTORE / ZINTERSTORE were using lookupKeyWrite.
      So there bugs are introduced in 6.2 and will be resolved before it is released.
      
      the implications of this bug are also not big:
      The sole difference between LookupKeyRead and LookupKeyWrite is for command executed on a replica, which are not received from its master client. (for the master, and for the master client on the replica, these two functions behave the same)!
      8a81ed1b
  30. 07 Jan, 2021 1 commit
    • Jonah H. Harris's avatar
      Add ZRANGESTORE command, and improve ZSTORE command (#7844) · b5029dfd
      Jonah H. Harris authored
      
      
      Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX].
      
      Syntax for the new ZRANGESTORE command:
      ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count]
      
      New syntax for ZRANGE:
      ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count]
      
      Old syntax for ZRANGE:
      ZRANGE [WITHSCORES]
      
      Other ZRANGE commands remain unchanged.
      
      The implementation uses common code for all of these, by utilizing a consumer interface that in one
      command response to the client, and in the other command stores a zset key.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b5029dfd
  31. 24 Dec, 2020 1 commit
  32. 09 Dec, 2020 1 commit
  33. 06 Dec, 2020 2 commits
    • Oran Agra's avatar
      Sanitize dump payload: validate no duplicate records in hash/zset/intset · 3716950c
      Oran Agra authored
      If RESTORE passes successfully with full sanitization, we can't affort
      to crash later on assertion due to duplicate records in a hash when
      converting it form ziplist to dict.
      This means that when doing full sanitization, we must make sure there
      are no duplicate records in any of the collections.
      3716950c
    • Wang Yuan's avatar
      Limit the main db and expires dictionaries to expand (#7954) · 75f9dec6
      Wang Yuan authored
      As we know, redis may reject user's requests or evict some keys if
      used memory is over maxmemory. Dictionaries expanding may make
      things worse, some big dictionaries, such as main db and expires dict,
      may eat huge memory at once for allocating a new big hash table and be
      far more than maxmemory after expanding.
      There are related issues: #4213 #4583
      
      More details, when expand dict in redis, we will allocate a new big
      ht[1] that generally is double of ht[0], The size of ht[1] will be
      very big if ht[0] already is big. For db dict, if we have more than
      64 million keys, we need to cost 1GB for ht[1] when dict expands.
      
      If the sum of used memory and new hash table of dict needed exceeds
      maxmemory, we shouldn't allow the dict to expand. Because, if we
      enable keys eviction, we still couldn't add much more keys after
      eviction and rehashing, what's worse, redis will keep less keys when
      redis only remains a little memory for storing new hash table instead
      of users' data. Moreover users can't write data in redis if disable
      keys eviction.
      
      What this commit changed ?
      
      Add a new member function expandAllowed for dict type, it provide a way
      for caller to allow expand or not. We expose two parameters for this
      function: more memory needed for expanding and dict current load factor,
      users can implement a function to make a decision by them.
      For main db dict and expires dict type, these dictionaries may be very
      big and cost huge memory for expanding, so we implement a judgement
      function: we can stop dict to expand provisionally if used memory will
      be over maxmemory after dict expands, but to guarantee the performance
      of redis, we still allow dict to expand if dict load factor exceeds the
      safe load factor.
      Add test cases to verify we don't allow main db to expand when left
      memory is not enough, so that avoid keys eviction.
      
      Other changes:
      
      For new hash table size when expand. Before this commit, the size is
      that double used of dict and later _dictNextPower. Actually we aim to
      control a dict load factor between 0.5 and 1.0. Now we replace *2 with
      +1, since the first check is that used >= size, the outcome of before
      will usually be the same as _dictNextPower(used+1). The only case where
      it'll differ is when dict_can_resize is false during fork, so that later
      the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e.
      _dictNextPower(1025*2) will return 4096).
      Fix rehash test cases due to changing algorithm of new hash table size
      when expand.
      75f9dec6
  34. 03 Dec, 2020 1 commit