1. 20 Jun, 2023 8 commits
    • Oran Agra's avatar
      Fix broken protocol when PUBLISH emits local push inside MULTI (#12326) · 8ad8f0f9
      Oran Agra authored
      When a connection that's subscribe to a channel emits PUBLISH inside MULTI-EXEC,
      the push notification messes up the EXEC response.
      
      e.g. MULTI, PING, PUSH foo bar, PING, EXEC
      the EXEC's response will contain: PONG, {message foo bar}, 1. and the second PONG
      will be delivered outside the EXEC's response.
      
      Additionally, this PR changes the order of responses in case of a plain PUBLISH (when
      the current client also subscribed to it), by delivering the push after the command's
      response instead of before it.
      This also affects modules calling RM_PublishMessage in a similar way, so that we don't
      run the risk of getting that push mixed together with the module command's response.
      8ad8f0f9
    • judeng's avatar
      use embedded string object and more efficient ll2string for long long value... · 93708c7f
      judeng authored
      
      use embedded string object and more efficient ll2string for long long value convert to string (#12250)
      
      A value of type long long is always less than 21 bytes when convert to a
      string, so always meets the conditions for using embedded string object
      which can always get memory reduction and performance gain (less calls
      to the heap allocator).
      Additionally, for the conversion of longlong type to sds, we also use a faster
      algorithm (the one in util.c instead of the one that used to be in sds.c). 
      
      For the DECR command on 32-bit Redis, we get about a 5.7% performance
      improvement. There will also be some performance gains for some commands
      that heavily use sdscatfmt to convert numbers, such as INFO.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      93708c7f
    • Binbin's avatar
      zrangeGenericCommand add check for negative offset (#9052) · d9c2ef8a
      Binbin authored
      Now we will check the offset in zrangeGenericCommand.
      With a negative offset, we will throw an error and return.
      
      This also resolve the issue of zeroing the destination key
      in case of the "store" variant when we input a negative offset.
      ```
      127.0.0.1:6379> set key value
      OK
      127.0.0.1:6379> zrangestore key myzset 0 10 byscore limit -1 10
      (integer) 0
      127.0.0.1:6379> exists key
      (integer) 0
      ```
      
      This change affects the following commands:
      - ZRANGE / ZRANGESTORE / ZRANGEBYLEX / ZRANGEBYSCORE
      - ZREVRANGE / ZREVRANGEBYSCORE / ZREVRANGEBYLEX
      d9c2ef8a
    • Binbin's avatar
      Fix ZRANK/ZREVRANK reply_schema description (#12331) · d306d861
      Binbin authored
      The parameter name is WITHSCORE instead of WITHSCORES.
      d306d861
    • mstmdev's avatar
      13e17e94
    • Wen Hui's avatar
      Sanitizer reported memory leak for '--invalid' option or port number is missed... · 813924b4
      Wen Hui authored
      
      Sanitizer reported memory leak for '--invalid' option or port number is missed cases to redis-server. (#12322)
      
      Observed that the sanitizer reported memory leak as clean up is not done
      before the process termination in negative/following cases:
      
      **- when we passed '--invalid' as option to redis-server.**
      
      ```
       -vm:~/mem-leak-issue/redis$ ./src/redis-server --invalid
      
      *** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
      Reading the configuration file, at line 2
      >>> 'invalid'
      Bad directive or wrong number of arguments
      
      =================================================================
      ==865778==ERROR: LeakSanitizer: detected memory leaks
      
      Direct leak of 8 byte(s) in 1 object(s) allocated from:
          #0 0x7f0985f65867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
          #1 0x558ec86686ec in ztrymalloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:117
          #2 0x558ec86686ec in ztrymalloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:135
          #3 0x558ec86686ec in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:276
          #4 0x558ec86686ec in zrealloc /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:327
          #5 0x558ec865dd7e in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1172
          #6 0x558ec87a1be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
          #7 0x558ec87a13b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
          #8 0x558ec85e6f15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
          #9 0x7f09856e5d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
      
      SUMMARY: AddressSanitizer: 8 byte(s) leaked in 1 allocation(s).
      
      ```
      
      **- when we pass '--port' as option and missed to add port number to redis-server.**
      
      ```
      vm:~/mem-leak-issue/redis$ ./src/redis-server --port
      
      *** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
      Reading the configuration file, at line 2
      >>> 'port'
      wrong number of arguments
      
      =================================================================
      ==865846==ERROR: LeakSanitizer: detected memory leaks
      
      Direct leak of 8 byte(s) in 1 object(s) allocated from:
          #0 0x7fdcdbb1f867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
          #1 0x557e8b04f6ec in ztrymalloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:117
          #2 0x557e8b04f6ec in ztrymalloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:135
          #3 0x557e8b04f6ec in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:276
          #4 0x557e8b04f6ec in zrealloc /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:327
          #5 0x557e8b044d7e in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1172
          #6 0x557e8b188be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
          #7 0x557e8b1883b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
          #8 0x557e8afcdf15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
          #9 0x7fdcdb29fd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
      
      Indirect leak of 10 byte(s) in 1 object(s) allocated from:
          #0 0x7fdcdbb1fc18 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
          #1 0x557e8b04f9aa in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:287
          #2 0x557e8b04f9aa in ztryrealloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:317
          #3 0x557e8b04f9aa in zrealloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:342
          #4 0x557e8b033f90 in _sdsMakeRoomFor /home/ubuntu/mem-leak-issue/redis/src/sds.c:271
          #5 0x557e8b033f90 in sdsMakeRoomFor /home/ubuntu/mem-leak-issue/redis/src/sds.c:295
          #6 0x557e8b033f90 in sdscatlen /home/ubuntu/mem-leak-issue/redis/src/sds.c:486
          #7 0x557e8b044e1f in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1165
          #8 0x557e8b188be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
          #9 0x557e8b1883b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
          #10 0x557e8afcdf15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
          #11 0x7fdcdb29fd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
      
      SUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s).
      
      ```
      
      As part analysis found that the sdsfreesplitres is not called when this condition checks are being hit.
      
      Output after the fix:
      
      
      ```
      vm:~/mem-leak-issue/redis$ ./src/redis-server --invalid
      
      *** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
      Reading the configuration file, at line 2
      >>> 'invalid'
      Bad directive or wrong number of arguments
      vm:~/mem-leak-issue/redis$
      
      ===========================================
      vm:~/mem-leak-issue/redis$ ./src/redis-server --jdhg
      
      *** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
      Reading the configuration file, at line 2
      >>> 'jdhg'
      Bad directive or wrong number of arguments
      
      ---------------------------------------------------------------------------
      vm:~/mem-leak-issue/redis$ ./src/redis-server --port
      
      *** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
      Reading the configuration file, at line 2
      >>> 'port'
      wrong number of arguments
      ```
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      813924b4
    • Shaya Potter's avatar
      Add ability for modules to know which client is being cmd filtered (#12219) · 07316f16
      Shaya Potter authored
      Adds API
      - RedisModule_CommandFilterGetClientId()
      
      Includes addition to commandfilter test module to validate that it works
      by performing the same command from 2 different clients
      07316f16
    • Binbin's avatar
      Fix cluster human_nodename Getter data loss in nodes.conf (#12325) · cd4f3e20
      Binbin authored
      auxHumanNodenameGetter limited to %.40s, since we did not limit the
      length of config cluster-announce-human-nodename, %.40s will cause
      nodename data loss (we will persist it in nodes.conf).
      
      Additional modified auxHumanNodenamePresent to use sdslen.
      cd4f3e20
  2. 19 Jun, 2023 1 commit
    • Binbin's avatar
      Optimize PSUBSCRIBE and PUNSUBSCRIBE from O(N*M) to O(N) (#12298) · b5106249
      Binbin authored
      In the original implementation, the time complexity of the commands
      is actually O(N*M), where N is the number of patterns the client is
      already subscribed and M is the number of patterns to subscribe to.
      The docs are all wrong about this.
      
      Specifically, because the original client->pubsub_patterns is a list,
      so we need to do listSearchKey which is O(N). In this PR, we change it
      to a dict, so the search becomes O(1).
      
      At the same time, both pubsub_channels and pubsubshard_channels are dicts.
      Changing pubsub_patterns to a dictionary improves the readability and
      maintainability of the code.
      b5106249
  3. 18 Jun, 2023 1 commit
    • Wen Hui's avatar
      Cluster human readable nodename feature (#9564) · 070453ee
      Wen Hui authored
      
      
      This PR adds a human readable name to a node in clusters that are visible as part of error logs. This is useful so that admins and operators of Redis cluster have better visibility into failures without having to cross-reference the generated ID with some logical identifier (such as pod-ID or EC2 instance ID). This is mentioned in #8948. Specific nodenames can be set by using the variable cluster-announce-human-nodename. The nodename is gossiped using the clusterbus extension in #9530.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      070453ee
  4. 16 Jun, 2023 3 commits
    • sundb's avatar
      Use Reservoir Sampling for random sampling of dict, and fix hang during fork (#12276) · b00a2351
      sundb authored
      
      
      ## Issue:
      When a dict has a long chain or the length of the chain is longer than
      the number of samples, we will never be able to sample the elements
      at the end of the chain using dictGetSomeKeys().
      This could mean that SRANDMEMBER can be hang in and endless loop.
      The most severe case, is the pathological case of when someone uses SCAN+DEL
      or SSCAN+SREM creating an unevenly distributed dict.
      This was amplified by the recent change in #11692 which prevented a
      down-sizing rehashing while there is a fork.
      
      ## Solution
      1. Before, we will stop sampling when we reach the maximum number
        of samples, even if there is more data after the current chain.
        Now when we reach the maximum we use the Reservoir Sampling
        algorithm to fairly sample the end of the chain that cannot be sampled
      2. Fix the rehashing code, so that the same as it allows rehashing for up-sizing
        during fork when the ratio is extreme, it will allow it for down-sizing as well.
      
      Issue was introduced (or became more severe) by #11692
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b00a2351
    • Binbin's avatar
      Fix SPOP/RESTORE propagation when doing lazy free (#12320) · 439b0315
      Binbin authored
      In SPOP, when COUNT is greater than or equal to set's size,
      we will remove the set. In dbDelete, we will do DEL or UNLINK
      according to the lazy flag. This is also required for propagate.
      
      In RESTORE, we won't store expired keys into the db, see #7472.
      When used together with REPLACE, it should emit a DEL or UNLINK
      according to the lazy flag.
      
      This PR also adds tests to cover the propagation. The RESTORE
      test will also cover #7472.
      439b0315
    • Binbin's avatar
      Add command being unblocked cause another command to get unblocked execution order test (#12324) · 9dc6f93e
      Binbin authored
      * Add command being unblocked cause another command to get unblocked execution order test
      
      In #12301, we observed that if the
      `while(listLength(server.ready_keys) != 0)`
      in handleClientsBlockedOnKeys is changed to
      `if(listLength(server.ready_keys) != 0)`,
      the order of command execution will change.
      
      It is wrong to change that. It means that if a command
      being unblocked causes another command to get unblocked
      (like a BLMOVE would do), then the new unblocked command
      will wait for later to get processed rather than right away.
      
      It'll not have any real implication if we change that since
      we do call handleClientsBlockedOnKeys in beforeSleep again,
      and redis will still behave correctly, but we don't change that.
      
      An example:
      1. $rd1 blmove src{t} dst{t} left right 0
      2. $rd2 blmove dst{t} src{t} right left 0
      3. $rd3 set key1{t}, $rd3 lpush src{t}, $rd3 set key2{t} in a pipeline
      
      The correct order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. lmove dst{t} src{t} right left
      5. set key2{t}
      
      The wrong order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. set key2{t}
      5. lmove dst{t} src{t} right left
      
      This PR adds corresponding test to cover it.
      
      * Add comment near while(listLength(server.ready_keys) != 0)
      9dc6f93e
  5. 15 Jun, 2023 2 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Prevent PEJ on loading and on readonly replica. (#12304) · cefe4566
      Meir Shpilraien (Spielrein) authored
      While Redis loading data from disk (AOF or RDB), modules will get
      key space notifications. In such stage the module should not register
      any PEJ, the main reason this is forbidden is that PEJ purpose is to
      perform a write operation as a reaction to the key space notification.
      Write operations should not be performed while loading data and so
      there is no reason to register a PEJ. 
      
      Same argument also apply to readonly replica. module should not
      perform any writes as a reaction to key space notifications and so it
      should not register a PEJ.
      
      If a module need to perform some other task which is not involve
      writing, he can do it on the key space notification callback itself.
      cefe4566
    • Binbin's avatar
      Optimize SET PXAT to reduce calls of rewriteClientCommandVector (#12316) · 25447553
      Binbin authored
      In PXAT case, there is no need to do the rewriteClientCommandVector,
      a simply benchmark show we gain a improvement of 10%.
      25447553
  6. 14 Jun, 2023 1 commit
    • judeng's avatar
      improve performance for keys with expiration time (#12177) · 789c33bb
      judeng authored
      This change only affects keys with expiry time.
      For SETEX, the average improvement is 5%, and for GET with
      expiation key, we gain a improvement of 13%.
      
      When keys have expiration time, Redis has an assertion to look
      up the main dict every time when it touches the expires.
      This comes with a performance const, especially during rehash.
      the damage will be double.
      
      It looks like that assert was added some ten years old, maybe out
      of paranoia, and there's probably no reason to keep it at that cost.
      789c33bb
  7. 13 Jun, 2023 2 commits
    • Harkrishn Patro's avatar
      Allow cluster slots/shards api to respond during loading (#12269) · a9e32767
      Harkrishn Patro authored
      It would be helpful for clients to get cluster slots/shards information during a node failover and is loading data.
      a9e32767
    • Binbin's avatar
      Fix XREADGROUP BLOCK stuck in endless loop (#12301) · e7129e43
      Binbin authored
      
      
      For the XREADGROUP BLOCK > scenario, there is an endless loop.
      Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command
      
      The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
      looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
      and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
      i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
      is problematic since the code keeps accessing the global list.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      e7129e43
  8. 12 Jun, 2023 1 commit
  9. 11 Jun, 2023 2 commits
    • YaacovHazan's avatar
      Reset command duration for rejected command. (#12247) · 0bfb6d55
      YaacovHazan authored
      In 7.2, After 971b177f
      
       we make sure (assert) that
      the duration has been recorded when resetting the client.
      
      This is not true for rejected commands.
      The use case I found is a blocking command that an ACL rule changed before
      it was unblocked, and while reprocessing it, the command rejected and triggered the assert.
      
      The PR reset the command duration inside rejectCommand / rejectCommandSds.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0bfb6d55
    • Binbin's avatar
      Fix ACLAppendUserForLoading memory leak when merge error (#12296) · 5fd9756d
      Binbin authored
      This leak will only happen in loadServerConfigFromString,
      that is, when we are loading a redis.conf, and the user is wrong.
      
      Because it happens in loadServerConfigFromString, redis will
      exit if there is an error, so this is actually just a cleanup.
      5fd9756d
  10. 08 Jun, 2023 1 commit
  11. 05 Jun, 2023 1 commit
  12. 30 May, 2023 1 commit
  13. 29 May, 2023 2 commits
  14. 28 May, 2023 5 commits
  15. 26 May, 2023 1 commit
    • Binbin's avatar
      Add new loglevel nothing to sentinel-config.json (#12235) · e775b34e
      Binbin authored
      It was missing in #12223, and the reply-schemas daily
      was failing:
      ```
      jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas
      
      Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
          {'oneOf': [{'const': 'debug'},
                     {'const': 'verbose'},
                     {'const': 'notice'},
                     {'const': 'warning'},
                     {'const': 'unknown'}]}
      
      On instance['loglevel']:
          'nothing'
      ```
      e775b34e
  16. 24 May, 2023 5 commits
    • mojh7's avatar
      Fix typo functoin -> function (#12218) · 627c610f
      mojh7 authored
      functoin -> function
      627c610f
    • Oran Agra's avatar
      Make a light weight version (default) of DEBUG HTSTATS (#12212) · 3ca451c4
      Oran Agra authored
      The light version only shows the table sizes, while the pre-existing
      version that shows chain length stats is reachable with the `full` argument.
      
      This should allow looking into rehashing state, even on huge dicts, on
      which we're afraid to run the command for fear of causing a server freeze.
      
      Also, fix a possible overflow in dictGetStats.
      3ca451c4
    • Oran Agra's avatar
      CI to validate commands.def is up to date (#12227) · c871db24
      Oran Agra authored
      and update recent SENTINEL CONFIG changes.
      c871db24
    • judeng's avatar
      postpone the initialization of oject's lru&lfu until it is added to the db as... · d71478a8
      judeng authored
      postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
      
      This pr can get two performance benefits:
      1. Stop redundant initialization when most robj objects are created
      2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
      
      Another code optimization:
      deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
      robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
      d71478a8
    • Binbin's avatar
      Sync the new loglevel nothing to sentinel (#12223) · d0994c5b
      Binbin authored
      We add a new loglevel 'nothing' to disable logging in #12133.
      This PR syncs that config change to sentinel. Because in #11214
      we support modifying loglevel in runtime.
      
      Although I think sentinel doesn't need this nothing config,
      it's better to be consistent.
      d0994c5b
  17. 23 May, 2023 3 commits
    • Ping Xie's avatar
      Exclude aux fields from "cluster nodes" and "cluster replicas" output (#12166) · 4c74dd98
      Ping Xie authored
      This commit excludes aux fields from the output of the `cluster nodes` and `cluster replicas` command.
      We may decide to re-introduce them in some form or another in the future, but not in v7.2.
      4c74dd98
    • zhaozhao.zz's avatar
      add a new loglevel 'nothing' to disable logging (#12133) · 07ea2204
      zhaozhao.zz authored
      Users can record logs of different levels by setting the `loglevel`.
      However, sometimes there are many logs even at the warning level,
      which can affect the performance of Redis.
      
      For example, when a user accesses the tls-port using a non-encrypted link,
      Redis will log lots of "# Error accepting a client connection: ...".
      
      We can provide the ability to disable logging so that users can temporarily turn
      off logging and turn it back on after the problem is resolved.
      07ea2204
    • Shaya Potter's avatar
      Fix memory leak when RM_Call's RUN_AS_USER fails (#12158) · 71e6abe4
      Shaya Potter authored
      previously the argv wasn't freed so would leak.  not a common case, but should be handled.
      
      Solution: move RUN_AS_USER setup and error exit to the right place.
      this way, when we do `goto cleanup` (instead of return) it'll automatically do the right thing (including autoMemoryAdd)
      Removed the user argument from moduleAllocTempClient (reverted to the state before 6e993a5d
      
      )
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      71e6abe4