1. 18 Jun, 2023 1 commit
    • Wen Hui's avatar
      Cluster human readable nodename feature (#9564) · 070453ee
      Wen Hui authored
      
      
      This PR adds a human readable name to a node in clusters that are visible as part of error logs. This is useful so that admins and operators of Redis cluster have better visibility into failures without having to cross-reference the generated ID with some logical identifier (such as pod-ID or EC2 instance ID). This is mentioned in #8948. Specific nodenames can be set by using the variable cluster-announce-human-nodename. The nodename is gossiped using the clusterbus extension in #9530.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      070453ee
  2. 16 Jun, 2023 4 commits
    • sundb's avatar
      Use Reservoir Sampling for random sampling of dict, and fix hang during fork (#12276) · b00a2351
      sundb authored
      
      
      ## Issue:
      When a dict has a long chain or the length of the chain is longer than
      the number of samples, we will never be able to sample the elements
      at the end of the chain using dictGetSomeKeys().
      This could mean that SRANDMEMBER can be hang in and endless loop.
      The most severe case, is the pathological case of when someone uses SCAN+DEL
      or SSCAN+SREM creating an unevenly distributed dict.
      This was amplified by the recent change in #11692 which prevented a
      down-sizing rehashing while there is a fork.
      
      ## Solution
      1. Before, we will stop sampling when we reach the maximum number
        of samples, even if there is more data after the current chain.
        Now when we reach the maximum we use the Reservoir Sampling
        algorithm to fairly sample the end of the chain that cannot be sampled
      2. Fix the rehashing code, so that the same as it allows rehashing for up-sizing
        during fork when the ratio is extreme, it will allow it for down-sizing as well.
      
      Issue was introduced (or became more severe) by #11692
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b00a2351
    • Binbin's avatar
      Fix SPOP/RESTORE propagation when doing lazy free (#12320) · 439b0315
      Binbin authored
      In SPOP, when COUNT is greater than or equal to set's size,
      we will remove the set. In dbDelete, we will do DEL or UNLINK
      according to the lazy flag. This is also required for propagate.
      
      In RESTORE, we won't store expired keys into the db, see #7472.
      When used together with REPLACE, it should emit a DEL or UNLINK
      according to the lazy flag.
      
      This PR also adds tests to cover the propagation. The RESTORE
      test will also cover #7472.
      439b0315
    • YaacovHazan's avatar
      Removing duplicated tests (#12318) · 5da9eecd
      YaacovHazan authored
      In 4ba47d2d the following tests added in both tracking.tcl and introspection.tcl
      
      - Coverage: Basic CLIENT CACHING
      - Coverage: Basic CLIENT REPLY
      - Coverage: Basic CLIENT TRACKINGINFO
      - Coverage: Basic CLIENT GETREDIR
      5da9eecd
    • Binbin's avatar
      Add command being unblocked cause another command to get unblocked execution order test (#12324) · 9dc6f93e
      Binbin authored
      * Add command being unblocked cause another command to get unblocked execution order test
      
      In #12301, we observed that if the
      `while(listLength(server.ready_keys) != 0)`
      in handleClientsBlockedOnKeys is changed to
      `if(listLength(server.ready_keys) != 0)`,
      the order of command execution will change.
      
      It is wrong to change that. It means that if a command
      being unblocked causes another command to get unblocked
      (like a BLMOVE would do), then the new unblocked command
      will wait for later to get processed rather than right away.
      
      It'll not have any real implication if we change that since
      we do call handleClientsBlockedOnKeys in beforeSleep again,
      and redis will still behave correctly, but we don't change that.
      
      An example:
      1. $rd1 blmove src{t} dst{t} left right 0
      2. $rd2 blmove dst{t} src{t} right left 0
      3. $rd3 set key1{t}, $rd3 lpush src{t}, $rd3 set key2{t} in a pipeline
      
      The correct order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. lmove dst{t} src{t} right left
      5. set key2{t}
      
      The wrong order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. set key2{t}
      5. lmove dst{t} src{t} right left
      
      This PR adds corresponding test to cover it.
      
      * Add comment near while(listLength(server.ready_keys) != 0)
      9dc6f93e
  3. 15 Jun, 2023 1 commit
  4. 14 Jun, 2023 1 commit
  5. 13 Jun, 2023 1 commit
    • Binbin's avatar
      Fix XREADGROUP BLOCK stuck in endless loop (#12301) · e7129e43
      Binbin authored
      
      
      For the XREADGROUP BLOCK > scenario, there is an endless loop.
      Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command
      
      The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
      looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
      and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
      i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
      is problematic since the code keeps accessing the global list.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      e7129e43
  6. 12 Jun, 2023 1 commit
  7. 11 Jun, 2023 3 commits
    • YaacovHazan's avatar
      Reset command duration for rejected command. (#12247) · 0bfb6d55
      YaacovHazan authored
      In 7.2, After 971b177f
      
       we make sure (assert) that
      the duration has been recorded when resetting the client.
      
      This is not true for rejected commands.
      The use case I found is a blocking command that an ACL rule changed before
      it was unblocked, and while reprocessing it, the command rejected and triggered the assert.
      
      The PR reset the command duration inside rejectCommand / rejectCommandSds.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0bfb6d55
    • Chen Tianjie's avatar
      Allow bigger tolerance in eventloop duration test. (#12179) · 2d7d3911
      Chen Tianjie authored
      In #11963, some new tests about eventloop duration were added, which includes time measurement in TCL scripts. This has caused some unexpected CI failures, such as #12169 and #12177, due to slow test servers or some performance jittering.
      2d7d3911
    • Wen Hui's avatar
      Adding missing test cases for Addslot Command (#12288) · d412269f
      Wen Hui authored
      Added missing test case coverage for below scenarios:
      
      1. The command only works if all the specified slots are, from
        the point of view of the node receiving the command, currently
        not assigned. A node will refuse to take ownership for slots that
        already belong to some other node (including itself).
      2. The command fails if the same slot is specified multiple times.
      d412269f
  8. 28 May, 2023 3 commits
    • Oran Agra's avatar
      Optimize MSETNX to avoid double lookup (#11944) · 2764dc37
      Oran Agra authored
      This is a redo of #11594 which got reverted in #11940
      It improves performance by avoiding double lookup of the the key.
      2764dc37
    • Oran Agra's avatar
      Fix WAIT for clients being blocked in a module command (#12220) · 6117f288
      Oran Agra authored
      So far clients being blocked and unblocked by a module command would
      update the c->woff variable and so WAIT was ineffective and got released
      without waiting for the command actions to propagate.
      
      This seems to have existed since forever, but not for RM_BlockClientOnKeys.
      
      It is problematic though to know if the module did or didn't propagate
      anything in that command, so for now, instead of adding an API, we'll
      just update the woff to the latest offset when unblocking, this will
      cause the client to possibly wait excessively, but that's not that bad.
      6117f288
    • Wen Hui's avatar
      [BUG] Incorrect error msg for XREAD command (#12238) · 1a188e4e
      Wen Hui authored
      
      
      XREAD only supports a special ID of $ and XREADGROUP only supports ^.
      make sure not to suggest the wrong one when rerunning an error about unbalanced ID arguments
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1a188e4e
  9. 24 May, 2023 2 commits
    • judeng's avatar
      postpone the initialization of oject's lru&lfu until it is added to the db as... · d71478a8
      judeng authored
      postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
      
      This pr can get two performance benefits:
      1. Stop redundant initialization when most robj objects are created
      2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
      
      Another code optimization:
      deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
      robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
      d71478a8
    • Wen Hui's avatar
      Adding test case for hvals, hkeys, hexists against wrong type (#12198) · d6648899
      Wen Hui authored
      HVALS, HKEYS and HEXISTS commands wrong type test cases were not covered so added the test cases.
      d6648899
  10. 23 May, 2023 1 commit
  11. 18 May, 2023 2 commits
    • Binbin's avatar
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and... · 48757934
      Binbin authored
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and expand dict in advance (#12185)
      
      For zsets that will eventually be stored as the skiplist encoding (has a dict),
      we can convert it to skiplist ahead of time. This change checks the number
      of arguments in the ZADD command, and converts the data-structure
      if the number of new entries exceeds the listpack-max-entries configuration.
      This can cause us to over-allocate memory if there are duplicate entries in the
      input, which is unexpected.
      
      For ZRANGESTORE, we know the size of the zset, so we can expand
      the dict in advance, to avoid the temporary dict from being rehashed
      while it grows.
      
      Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
      48757934
    • Hanna Fadida's avatar
      Add BITFIELD_RO basic tests for non-repl use cases (#12187) · 37cf1984
      Hanna Fadida authored
      Current tests for BITFIELD_RO command are skipped in the external mode,
      and therefore reply-schemas-validator reports a coverage error.
      This PR adds basic tests to increase coverage.
      37cf1984
  12. 16 May, 2023 2 commits
    • Binbin's avatar
      Fix for set max entries edge case in setTypeCreate / setTypeMaybeConvert (#12183) · fd566f40
      Binbin authored
      In the judgment in setTypeCreate, we should judge size_hint <= max_entries.
      
      This results in the following inconsistencies:
      ```
      127.0.0.1:6379> config set set-max-intset-entries 5 set-max-listpack-entries 5
      OK
      
      127.0.0.1:6379> sadd intset_set1 1 2 3 4 5
      (integer) 5
      127.0.0.1:6379> object encoding intset_set1
      "hashtable"
      127.0.0.1:6379> sadd intset_set2 1 2 3 4
      (integer) 4
      127.0.0.1:6379> sadd intset_set2 5
      (integer) 1
      127.0.0.1:6379> object encoding intset_set2
      "intset"
      
      127.0.0.1:6379> sadd listpack_set1 a 1 2 3 4
      (integer) 5
      127.0.0.1:6379> object encoding listpack_set1
      "hashtable"
      127.0.0.1:6379> sadd listpack_set2 a 1 2 3
      (integer) 4
      127.0.0.1:6379> sadd listpack_set2 4
      (integer) 1
      127.0.0.1:6379> object encoding listpack_set2
      "listpack"
      ```
      
      This was introduced in #12019, added corresponding tests.
      fd566f40
    • Wen Hui's avatar
      Adding missing test case for smembers scard commands (#12148) · e4527288
      Wen Hui authored
      Minor missing test case addition.
      
      SMEMBERS SCARD against non set
      SMEMBERS SCARD against non existing key
      e4527288
  13. 14 May, 2023 2 commits
  14. 12 May, 2023 1 commit
    • Chen Tianjie's avatar
      Add basic eventloop latency measurement. (#11963) · 29ca8795
      Chen Tianjie authored
      The measured latency(duration) includes the list below, which can be shown by `INFO STATS`.
      ```
      eventloop_cycles  // ever increasing counter
      eventloop_duration_sum // cumulative duration of eventloop in microseconds
      eventloop_duration_cmd_sum  // cumulative duration of executing commands in microseconds
      instantaneous_eventloop_cycles_per_sec  // average eventloop count per second in recent 1.6s
      instantaneous_eventloop_duration_usec  // average single eventloop duration in recent 1.6s
      ```
      
      Also added some experimental metrics, which are shown only when `INFO DEBUG` is called.
      This section isn't included in the default INFO, or even in `INFO ALL` and the fields in this section
      can change in the future without considering backwards compatibility.
      ```
      eventloop_duration_aof_sum  // cumulative duration of writing AOF
      eventloop_duration_cron_sum  // cumulative duration cron jobs (serverCron, beforeSleep excluding IO and AOF)
      eventloop_cmd_per_cycle_max  // max number of commands executed in one eventloop
      eventloop_duration_max  // max duration of one eventloop
      ```
      
      All of these are being reset by CONFIG RESETSTAT
      29ca8795
  15. 09 May, 2023 1 commit
    • Binbin's avatar
      EXPIRE precision test more attempts and more tolerant (#12150) · 6ab2174d
      Binbin authored
      The test failed on MacOS:
      ```
      *** [err]: EXPIRE precision is now the millisecond in tests/unit/expire.tcl
      Expected 'somevalue {}' to equal or match '{} {}'
      ```
      
      `set a [r get x]`, even though we tried 10 times, sometimes we
      still get {}, this is a time-sensitive test.
      
      In this PR, we add the following changes:
      1. More attempts, change it from 10 to 30.
      2. More tolerant, change the `after 900` to `after 800`.
      
      In addition, we judging $a in advance and changing `after 1100`
      to `after 300`, this will save us some times.
      6ab2174d
  16. 08 May, 2023 1 commit
  17. 07 May, 2023 2 commits
  18. 04 May, 2023 2 commits
    • Binbin's avatar
      Pause cron to prevent premature shrinking in querybuf test (#12126) · e49c2a52
      Binbin authored
      Tests occasionally fail since #12000:
      ```
      *** [err]: query buffer resized correctly when not idle in tests/unit/querybuf.tcl
      Expected 0 > 32768 (context: type eval line 11 cmd {assert {$orig_test_client_qbuf > 32768}} proc ::test)
      
      *** [err]: query buffer resized correctly with fat argv in tests/unit/querybuf.tcl
      query buffer should not be resized when client idle time smaller than 2s
      ```
      
      The reason may be because we set hz to 100, querybuf shrinks before we count
      client_query_buffer. We avoid this problem by setting pause-cron to 1.
      e49c2a52
    • guybe7's avatar
      multi.tcl: reset readraw at the end of the test (#12123) · 857c09b0
      guybe7 authored
      1. reset the readraw mode after a test that uses it. undetected since the
        only test after that on the same server didn't read any replies.
      2. fix a cross slot issue that was undetected in cluster mode because
        readraw doesn't throw exceptions on errors.
      857c09b0
  19. 01 May, 2023 1 commit
  20. 25 Apr, 2023 1 commit
  21. 19 Apr, 2023 1 commit
  22. 18 Apr, 2023 2 commits
    • Binbin's avatar
      Tests: Do not save an RDB by default and add a SIGTERM default AOFRW test (#12064) · 20533cc1
      Binbin authored
      
      
      In order to speed up tests, avoid saving an RDB (mostly notable on shutdown),
      except for tests that explicitly test the RDB mechanism
      
      In addition, use `shutdown-on-sigterm force` to prevetn shutdown from failing
      in case the server is in the middle of the initial AOFRW
      
      Also a a test that checks that the `shutdown-on-sigterm default` is to refuse
      shutdown if there's an initial AOFRW
      Co-authored-by: default avatarGuy Benoish <guy.benoish@redislabs.com>
      20533cc1
    • Wen Hui's avatar
      Added getrange missing testcase (#12061) · d2db4aa7
      Wen Hui authored
      Minor test case addition.
      Currently GETRANGE command does not have the test case coverage for the scenarios:
      An error is returned when key exists but of different type
      Added missing test cases for getrange command.
      d2db4aa7
  23. 16 Apr, 2023 1 commit
    • judeng's avatar
      avoid incorrect shrinking of querybuf when client is reading a big argv (#12000) · e7f18432
      judeng authored
      this pr fix two wrongs:
      1. When client’s querybuf is pre-allocated for a fat argv, we need to update the
        querybuf_peak of the client immediately to completely avoid the unexpected
        shrinking of querybuf in the next clientCron (before data arrives to set the peak).
      2. the protocol's bulklen does not include `\r\n`, but the allocation and the data we
        read does. so in `clientsCronResizeQueryBuffer`, the `resize` or `querybuf_peak`
        should add these 2 bytes.
      
      the first bug is likely to hit us on large payloads over slow connections, in which case
      transferring the payload can take longer and a cron event will be triggered (specifically
      if there are not a lot of clients)
      e7f18432
  24. 13 Apr, 2023 2 commits
    • Wen Hui's avatar
      Adding missing test cases for substring (#12039) · 4375b01c
      Wen Hui authored
      There is are some missing test cases for SUBSTR command.
      These might already be covered by GETRANGE, but no harm in adding them since they're simple.
      
      Added 3 test case.
      
      * start > stop
      * start and stop both greater than string length
      * when no key is present.
      4375b01c
    • Wen Hui's avatar
      Adding missing test cases for linsert command (#12040) · bc82309c
      Wen Hui authored
      Currently LINSERT command does not have the test case coverage for following scenarios.
      1. When key does not exist, it is considered an empty list and no operation is performed.
      2. An error is returned when key exists but does not hold a list value.
      
      Added above two missing test cases for linsert command.
      bc82309c
  25. 12 Apr, 2023 1 commit
    • Binbin's avatar
      Add RM_ReplyWithErrorFormat that can support format (#11923) · bfec2d70
      Binbin authored
      * Add RM_ReplyWithErrorFormat that can support format
      
      Reply with the error create from a printf format and arguments.
      
      If the error code is already passed in the string 'fmt', the error
      code provided is used, otherwise the string "-ERR " for the generic
      error code is automatically added.
      
      The usage is, for example:
          RedisModule_ReplyWithErrorFormat(ctx, "An error: %s", "foo");
          RedisModule_ReplyWithErrorFormat(ctx, "-WRONGTYPE Wrong Type: %s", "foo");
      
      The function always returns REDISMODULE_OK.
      bfec2d70