1. 06 Sep, 2023 3 commits
    • bodong.ybd's avatar
      Fix sort_ro get-keys function return wrong key number (#12522) · 0f14d327
      bodong.ybd authored
      Before:
      ```
      127.0.0.1:6379> command getkeys sort_ro key
      (empty array)
      127.0.0.1:6379>
      ```
      After:
      ```
      127.0.0.1:6379> command getkeys sort_ro key
      1) "key"
      127.0.0.1:6379>
      ```
      
      (cherry picked from commit b59f53ef)
      0f14d327
    • zhaozhao.zz's avatar
      do not call handleClientsBlockedOnKeys inside yielding command (#12459) · 4d67bb6a
      zhaozhao.zz authored
      
      
      Fix the assertion when a busy script (timeout) signal ready keys (like LPUSH),
      and then an arbitrary client's `allow-busy` command steps into `handleClientsBlockedOnKeys`
      try wake up clients blocked on keys (like BLPOP).
      
      Reproduction process:
      1. start a redis with aof
          `./redis-server --appendonly yes`
      2. exec blpop
          `127.0.0.1:6379> blpop a 0`
      3. use another client call a busy script and this script push the blocked key
          `127.0.0.1:6379> eval "redis.call('lpush','a','b') while(1) do end" 0`
      4. user a new client call an allow-busy command like auth
          `127.0.0.1:6379> auth a`
      
      BTW, this issue also break the atomicity of script.
      
      This bug has been around for many years, the old versions only have the
      atomic problem, only 7.0/7.2 has the assertion problem.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 8226f39f)
      4d67bb6a
    • sundb's avatar
      Skip test for sdsRemoveFreeSpace when mem_allocator is not jemalloc (#11878) · 646069a9
      sundb authored
      Test `trim on SET with big value` (introduced from #11817) fails under mac m1 with libc mem_allocator.
      The reason is that malloc(33000) will allocate 65536 bytes(>42000).
      This test still passes under ubuntu with libc mem_allocator.
      
      ```
      *** [err]: trim on SET with big value in tests/unit/type/string.tcl
      Expected [r memory usage key] < 42000 (context: type source line 471 file /Users/iospack/data/redis_fork/tests/unit/type/string.tcl cmd {assert {[r memory usage key] < 42000}} proc ::test)
      ```
      
      simple test under mac m1 with libc mem_allocator:
      ```c
      void *p = zmalloc(33000);
      printf("malloc size: %zu\n", zmalloc_size(p));
      
      # output
      malloc size: 65536
      ```
      
      (cherry picked from commit 3fba3ccd)
      646069a9
  2. 10 Jul, 2023 4 commits
    • Lior Lahav's avatar
      Fix possible crash in command getkeys (#12380) · bd1dac0c
      Lior Lahav authored
      
      
      When getKeysUsingKeySpecs processes a command with more than one key-spec,
      and called with a total of more than 256 keys, it'll call getKeysPrepareResult again,
      but since numkeys isn't updated, getKeysPrepareResult will not bother to copy key
      names from the old result (leaving these slots uninitialized). Furthermore, it did not
      consider the keys it already found when allocating more space.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b7559d9f)
      bd1dac0c
    • sundb's avatar
      Use Reservoir Sampling for random sampling of dict, and fix hang during fork (#12276) · 25f610fc
      sundb authored
      
      
      ## Issue:
      When a dict has a long chain or the length of the chain is longer than
      the number of samples, we will never be able to sample the elements
      at the end of the chain using dictGetSomeKeys().
      This could mean that SRANDMEMBER can be hang in and endless loop.
      The most severe case, is the pathological case of when someone uses SCAN+DEL
      or SSCAN+SREM creating an unevenly distributed dict.
      This was amplified by the recent change in #11692 which prevented a
      down-sizing rehashing while there is a fork.
      
      ## Solution
      1. Before, we will stop sampling when we reach the maximum number
        of samples, even if there is more data after the current chain.
        Now when we reach the maximum we use the Reservoir Sampling
        algorithm to fairly sample the end of the chain that cannot be sampled
      2. Fix the rehashing code, so that the same as it allows rehashing for up-sizing
        during fork when the ratio is extreme, it will allow it for down-sizing as well.
      
      Issue was introduced (or became more severe) by #11692
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b00a2351)
      25f610fc
    • Oran Agra's avatar
      Fix WAIT for clients being blocked in a module command (#12220) · 2ba8de9d
      Oran Agra authored
      So far clients being blocked and unblocked by a module command would
      update the c->woff variable and so WAIT was ineffective and got released
      without waiting for the command actions to propagate.
      
      This seems to have existed since forever, but not for RM_BlockClientOnKeys.
      
      It is problematic though to know if the module did or didn't propagate
      anything in that command, so for now, instead of adding an API, we'll
      just update the woff to the latest offset when unblocking, this will
      cause the client to possibly wait excessively, but that's not that bad.
      
      (cherry picked from commit 6117f288)
      2ba8de9d
    • Oran Agra's avatar
      Lua cjson and cmsgpack integer overflow issues (CVE-2022-24834) · f6a7c9f9
      Oran Agra authored
      
      
      * Fix integer overflows due to using wrong integer size.
      * Add assertions / panic when overflow still happens.
      * Deletion of dead code to avoid need to maintain it
      * Some changes are not because of bugs, but rather paranoia.
      * Improve cmsgpack and cjson test coverage.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      f6a7c9f9
  3. 17 Apr, 2023 3 commits
  4. 20 Mar, 2023 3 commits
  5. 28 Feb, 2023 9 commits
    • Oran Agra's avatar
      Integer Overflow in RAND commands can lead to assertion (CVE-2023-25155) · 2a2a582e
      Oran Agra authored
      Issue happens when passing a negative long value that greater than
      the max positive value that the long can store.
      2a2a582e
    • Tom Levy's avatar
      String pattern matching had exponential time complexity on pathological patterns (CVE-2022-36021) · 08255525
      Tom Levy authored
      Authenticated users can use string matching commands with a
      specially crafted pattern to trigger a denial-of-service attack on Redis,
      causing it to hang and consume 100% CPU time.
      08255525
    • ranshid's avatar
      Fix possible memory corruption in FLUSHALL when a client watches more than one key (#11854) · 7091b495
      ranshid authored
      
      
      Avoid calling unwatchAllKeys when running touchAllWatchedKeysInDb (which was unnecessary)
      This can potentially lead to use-after-free and memory corruption when the next entry
      pointer held by the watched keys iterator is freed when unwatching all keys of a specific client.
      found with address sanitizer, added a test which will not always fail (depending on the random
      dict hashing seed)
      problem introduced in #9829 (Reids 7.0)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 18017df7)
      7091b495
    • Madelyn Olson's avatar
    • judeng's avatar
      add test case and comments for active expiry in the writeable replica (#11789) · feb796d3
      judeng authored
      This test case is to cover a edge scenario: when a writable replica enabled AOF
      at the same time, active expiry keys which was created in writable replicas should
      propagate to the AOF file, and some versions might crash (fixed by #11615).
      For details, please refer to #11778
      
      (cherry picked from commit 40659c34)
      feb796d3
    • guybe7's avatar
      SCAN/RANDOMKEY and lazy-expire (#11788) · 2db72059
      guybe7 authored
      Starting from Redis 7.0 (#9890) we started wrapping everything a command
       propagates with MULTI/EXEC. The problem is that both SCAN and RANDOMKEY can
      lazy-expire arbitrary keys (similar behavior to active-expire), and put DELs in a transaction.
      
      Fix: When these commands are called without a parent exec-unit (e.g. not in EVAL or
      MULTI) we avoid wrapping their DELs in a transaction (for the same reasons active-expire
      and eviction avoids a transaction)
      
      This PR adds a per-command flag that indicates that the command may touch arbitrary
      keys (not the ones in the arguments), and uses that flag to avoid the MULTI-EXEC.
      For now, this flag is internal, since we're considering other solutions for the future.
      
      Note for cluster mode: if SCAN/RANDOMKEY is inside EVAL/MULTI it can still cause the
      same situation (as it always did), but it won't cause a CROSSSLOT because replicas and AOF
      do not perform slot checks.
      The problem with the above is mainly for 3rd party ecosystem tools that propagate commands
      from master to master, or feed an AOF file with redis-cli into a master.
      This PR aims to fix the regression in redis 7.0, and we opened #11792 to try to handle the
      bigger problem with lazy expire better for another release.
      
      (cherry picked from commit fd82bccd)
      2db72059
    • Harkrishn Patro's avatar
      Propagate message to a node only if the cluster link is healthy. (#11752) · ca0b6cae
      Harkrishn Patro authored
      Currently while a sharded pubsub message publish tries to propagate the message across the cluster, a NULL check is missing for clusterLink. clusterLink could be NULL if the link is causing memory beyond the set threshold cluster-link-sendbuf-limit and server terminates the link.
      
      This change introduces two things:
      
      Avoids the engine crashes on the publishing node if a message is tried to be sent to a node and the link is NULL.
      Adds a debugging tool CLUSTERLINK KILL to terminate the clusterLink between two nodes.
      
      (cherry picked from commit fd397568)
      ca0b6cae
    • uriyage's avatar
      Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766) · af80a4a5
      uriyage authored
      
      
      In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation
      size in order to:
      
      > reduce the need for realloc calls by making the sds implicitly take over
      the internal fragmentation
      
      This change was done most sds functions, excluding `sdsRemoveFreeSpace` and
      `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer)
      we call sdsRemoveFreeSpace when we see excessive free space and want to trim it.
      so if we don't trim it exactly to size, the caller may still see excessive free space and
      call it again and again.
      
      However, this resulted in some excessive calls to realloc, even when there's no need
      and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13).
      
      It turns out that a call for realloc with jemalloc can be expensive even if it ends up
      doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid
      the call for realloc.
      
      in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common
      code. the difference between them was that sdsResize would avoid using SDS_TYPE_5,
      since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace
      would permit using SDS_TYPE_5 and get an optimal memory consumption.
      now both methods take a `would_regrow` argument that makes it more explicit.
      
      the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize
      and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both.
      
      The new test that was added to cover this concern used to pass before this PR as well,
      this PR is just a performance optimization and cleanup.
      
      Benchmark:
      `redis-benchmark -c 100 -t set  -d 512 -P 10  -n  100000000`
      on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs).
      some 4.5% improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 46393f98)
      af80a4a5
    • Wen Hui's avatar
      Fix command BITFIELD_RO and BITFIELD argument json file, add some test cases for them (#11445) · 4d5a4e4b
      Wen Hui authored
      According to the source code, the commands can be executed with only key name,
      and no GET/SET/INCR operation arguments.
      change the docs to reflect that by marking these arguments as optional.
      also add tests.
      
      (cherry picked from commit fea9bbbe)
      4d5a4e4b
  6. 16 Jan, 2023 5 commits
    • Oran Agra's avatar
      Fix range issues in ZRANDMEMBER and HRANDFIELD (CVE-2023-22458) · 3f1f0203
      Oran Agra authored
      missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due
      to protocol limitations
      3f1f0203
    • Oran Agra's avatar
      Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) · 6c25c6b7
      Oran Agra authored
      Authenticated users issuing specially crafted SETRANGE and SORT(_RO)
      commands can trigger an integer overflow, resulting with Redis attempting
      to allocate impossible amounts of memory and abort with an OOM panic.
      6c25c6b7
    • Oran Agra's avatar
      Obuf limit, exit during loop in *RAND* commands and KEYS · 4537830e
      Oran Agra authored
      Related to the hang reported in #11671
      Currently, redis can disconnect a client due to reaching output buffer limit,
      it'll also avoid feeding that output buffer with more data, but it will keep
      running the loop in the command (despite the client already being marked for
      disconnection)
      
      This PR is an attempt to mitigate the problem, specifically for commands that
      are easy to abuse, specifically: KEYS, HRANDFIELD, SRANDMEMBER, ZRANDMEMBER.
      The RAND family of commands can take a negative COUNT argument (which is not
      bound to the number of elements in the key), so it's enough to create a key
      with one field, and then these commands can be used to hang redis.
      For KEYS the caller can use the existing keyspace in redis (if big enough).
      4537830e
    • Gabi Ganam's avatar
      Blocking command with a 0.001 seconds timeout blocks indefinitely (#11688) · 574a49b9
      Gabi Ganam authored
      Any value in the range of [0-1) turns to 0 when being cast from double to long long. This change rounds up instead of down for values that can't be stored precisely as long doubles.
      
      (cherry picked from commit eef29b68)
      574a49b9
    • Oran Agra's avatar
      Fix potential issue with Lua argv caching, module command filter and libc realloc (#11652) · 61a1d454
      Oran Agra authored
      TLDR: solve a problem introduced in Redis 7.0.6 (#11541) with
      RM_CommandFilterArgInsert being called from scripts, which can
      lead to memory corruption.
      
      Libc realloc can return the same pointer even if the size was changed. The code in
      freeLuaRedisArgv had an assumption that if the pointer didn't change, then the
      allocation didn't change, and the cache can still be reused.
      However, if rewriteClientCommandArgument or RM_CommandFilterArgInsert were
      used, it could be that we realloced the argv array, and the pointer didn't change, then
      a consecutive command being executed from Lua can use that argv cache reaching
      beyond its size.
      This was actually only possible with modules, since the decision to realloc was based
      on argc, rather than argv_len.
      
      (cherry picked from commit c8052122)
      61a1d454
  7. 16 Dec, 2022 1 commit
  8. 12 Dec, 2022 12 commits
    • Binbin's avatar
      Fix redis-cli cluster add-node race in cli.tcl (#11349) · 3c525fab
      Binbin authored
      There is a race condition in the test:
      ```
      *** [err]: redis-cli --cluster add-node with cluster-port in tests/unit/cluster/cli.tcl
      Expected '5' to be equal to '4' {assert_equal 5 [CI 0 cluster_known_nodes]} proc ::test)
      ```
      
      When using cli to add node, there can potentially be a race condition
      in which all nodes presenting cluster state o.k even though the added
      node did not yet meet all cluster nodes.
      
      This comment and the fix were taken from #11221. Also apply it in several
      other similar places.
      
      (cherry picked from commit a549b78c)
      3c525fab
    • ranshid's avatar
      fix test Migrate the last slot away from a node using redis-cli (#11221) · e1557e6c
      ranshid authored
      When using cli to add node, there can potentially be a race condition in
      which all nodes presenting cluster state o.k even though the added node
      did not yet meet all cluster nodes.
      this adds another utility function to wait until all cluster nodes see the same cluster size
      
      (cherry picked from commit c0ce97fa)
      e1557e6c
    • Binbin's avatar
      Fix timing issue in cluster test (#11008) · b5784fea
      Binbin authored
      A timing issue like this was reported in freebsd daily CI:
      ```
      *** [err]: Sanity test push cmd after resharding in tests/unit/cluster/cli.tcl
      Expected 'CLUSTERDOWN The cluster is down' to match '*MOVED*'
      ```
      
      We additionally wait for each node to reach a consensus on the cluster
      state in wait_for_condition to avoid the cluster down error.
      
      The fix just like #10495, quoting madolson's comment:
      Cluster check just verifies the the config state is self-consistent,
      waiting for cluster_state to be okay is an independent check that all
      the nodes actually believe each other are healthy.
      
      At the same time i noticed that unit/moduleapi/cluster.tcl has an exact
      same test, may have the same problem, also modified it.
      
      (cherry picked from commit 5ce64ab0)
      b5784fea
    • Binbin's avatar
      Fix CLUSTERDOWN issue in cluster reshard unblock test (#11139) · a43c51b2
      Binbin authored
      change the cluster-node-timeout from 1 to 1000
      
      (cherry picked from commit 3a16ad30)
      a43c51b2
    • Harkrishn Patro's avatar
      Optimize client memory usage tracking operation while client eviction is disabled (#11348) · 05c9378b
      Harkrishn Patro authored
      
      
      ## Issue
      During the client input/output buffer processing, the memory usage is
      incrementally updated to keep track of clients going beyond a certain
      threshold `maxmemory-clients` to be evicted. However, this additional
      tracking activity leads to unnecessary CPU cycles wasted when no
      client-eviction is required. It is applicable in two cases.
      
      * `maxmemory-clients` is set to `0` which equates to no client eviction
        (applicable to all clients)
      * `CLIENT NO-EVICT` flag is set to `ON` which equates to a particular
        client not applicable for eviction.
      
      ## Solution
      * Disable client memory usage tracking during the read/write flow when
        `maxmemory-clients` is set to `0` or `client no-evict` is `on`.
        The memory usage is tracked only during the `clientCron` i.e. it gets
        periodically updated.
      * Cleanup the clients from the memory usage bucket when client eviction
        is disabled.
      * When the maxmemory-clients config is enabled or disabled at runtime,
        we immediately update the memory usage buckets for all clients (tested
        scanning 80000 took some 20ms)
      
      Benchmark shown that this can improve performance by about 5% in
      certain situations.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit c0267b3f)
      05c9378b
    • filipe oliveira's avatar
      Reintroduce lua argument cache in luaRedisGenericCommand removed in v7.0 (#11541) · 6a6d8806
      filipe oliveira authored
      This mechanism aims to reduce calls to malloc and free when
      preparing the arguments the script sends to redis commands.
      This is a mechanism was originally implemented in 48c49c48
      and 4f686555
      
      , and was removed in #10220 (thinking it's not needed
      and that it has no impact), but it now turns out it was wrong, and it
      indeed provides some 5% performance improvement.
      
      The implementation is a little bit too simplistic, it assumes consecutive
      calls use the same size in the same arg index, but that's arguably
      sufficient since it's only aimed at caching very small things.
      
      We could even consider always pre-allocating args to the full
      LUA_CMD_OBJCACHE_MAX_LEN (64 bytes) rather than the right size for the argument,
      that would increase the chance they'll be able to be re-used.
      But in some way this is already happening since we're using
      sdsalloc, which in turn uses s_malloc_usable and takes ownership
      of the full side of the allocation, so we are padded to the allocator
      bucket size.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      (cherry picked from commit 2d80cd78)
      6a6d8806
    • filipe oliveira's avatar
      Speedup GEODIST with fixedpoint_d2string as an optimized version of snprintf %.4f (#11552) · 77570e39
      filipe oliveira authored
      
      
      GEODIST used snprintf("%.4f") for the reply using addReplyDoubleDistance,
      which was slow. This PR optimizes it without breaking compatibility by following
      the approach of ll2string with some changes to match the use case of distance
      and precision. I.e. we multiply it by 10000 format it as an integer, and then add
      a decimal point. This can achieve about 35% increase in the achievable ops/sec.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 61c85a2b)
      77570e39
    • DevineLiu's avatar
      [BUG] Fix announced ports not updating on local node when updated at runtime (#10745) · 81e72e8e
      DevineLiu authored
      
      
      The cluster-announce-port/cluster-announce-bus-port/cluster-announce-tls-port should take effect at runtime
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      (cherry picked from commit 25ffa79b)
      81e72e8e
    • Madelyn Olson's avatar
      Explicitly send function commands to monitor (#11510) · a26ac7eb
      Madelyn Olson authored
      Both functions and eval are marked as "no-monitor", since we want to explicitly feed in the script command before the commands generated by the script. Note that we want this behavior generally, so that commands can redact arguments before being added to the monitor.
      
      (cherry picked from commit d136bf28)
      a26ac7eb
    • uriyage's avatar
      Module CLIENT_CHANGE, Fix crash on free blocked client with DB!=0 (#11500) · 7ad786db
      uriyage authored
      
      
      In moduleFireServerEvent we change the real client DB to 0 on freeClient in case the event is REDISMODULE_EVENT_CLIENT_CHANGE.
      It results in a crash if the client is blocked on a key on other than DB 0.
      
      The DB change is not necessary even for module-client, as we set its DB to 0 on either createClient or moduleReleaseTempClient.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      (cherry picked from commit e4eb18b3)
      7ad786db
    • sundb's avatar
      Fix crash due to to reuse iterator entry after list deletion in module (#11383) · 9fc20f4f
      sundb authored
      
      
      In the module, we will reuse the list iterator entry for RM_ListDelete, but `listTypeDelete` will only update
      `quicklistEntry->zi` but not `quicklistEntry->node`, which will result in `quicklistEntry->node` pointing to
      a freed memory address if the quicklist node is deleted.
      
      This PR sync `key->u.list.index` and `key->u.list.entry` to list iterator after `RM_ListDelete`.
      
      This PR also optimizes the release code of the original list iterator.
      Co-authored-by: default avatarViktor Söderqvist <viktor@zuiderkwast.se>
      (cherry picked from commit 6dd21355)
      9fc20f4f
    • C Charles's avatar
      MIGTATE with AUTH that contains "keys" is getting wrong key names in... · f95af778
      C Charles authored
      MIGTATE with AUTH that contains "keys" is getting wrong key names in migrateGetKeys, leads to ACL errors (#11253)
      
      When using the MIGRATE, with a destination Redis that has the user name or password set to the string "keys",
      Redis would have determine the wrong set of key names the command is gonna access.
      This lead to ACL returning wrong authentication result.
      
      Destination instance:
      ```
      127.0.0.1:6380> acl setuser default >keys
      OK
      127.0.0.1:6380> acl setuser keys on nopass ~* &* +@all
      OK
      ```
      
      Source instance:
      ```
      127.0.0.1:6379> set a 123
      OK
      127.0.0.1:6379> acl setuser cc on nopass ~a* +@all
      OK
      127.0.0.1:6379> auth cc 1
      OK
      127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth keys keys a
      (error) NOPERM this user has no permissions to access one of the keys used as arguments
      127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth2 keys pswd keys a
      (error) NOPERM this user has no permissions to access one of the keys used as arguments
      ```
      
      Using `acl dryrun` we know that the parameters of `auth` and `auth2` are mistaken for the `keys` option.
      ```
      127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth keys keys a
      "This user has no permissions to access the 'keys' key"
      127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth2 keys pswd keys a
      "This user has no permissions to access the 'pswd' key"
      ```
      
      Fix the bug by editing db.c/migrateGetKeys function, which finds the `keys` option and all the keys following.
      
      (cherry picked from commit 9ab873d9)
      f95af778