1. 16 Jun, 2023 2 commits
    • YaacovHazan's avatar
      Removing duplicated tests (#12318) · 5da9eecd
      YaacovHazan authored
      In 4ba47d2d the following tests added in both tracking.tcl and introspection.tcl
      
      - Coverage: Basic CLIENT CACHING
      - Coverage: Basic CLIENT REPLY
      - Coverage: Basic CLIENT TRACKINGINFO
      - Coverage: Basic CLIENT GETREDIR
      5da9eecd
    • Binbin's avatar
      Add command being unblocked cause another command to get unblocked execution order test (#12324) · 9dc6f93e
      Binbin authored
      * Add command being unblocked cause another command to get unblocked execution order test
      
      In #12301, we observed that if the
      `while(listLength(server.ready_keys) != 0)`
      in handleClientsBlockedOnKeys is changed to
      `if(listLength(server.ready_keys) != 0)`,
      the order of command execution will change.
      
      It is wrong to change that. It means that if a command
      being unblocked causes another command to get unblocked
      (like a BLMOVE would do), then the new unblocked command
      will wait for later to get processed rather than right away.
      
      It'll not have any real implication if we change that since
      we do call handleClientsBlockedOnKeys in beforeSleep again,
      and redis will still behave correctly, but we don't change that.
      
      An example:
      1. $rd1 blmove src{t} dst{t} left right 0
      2. $rd2 blmove dst{t} src{t} right left 0
      3. $rd3 set key1{t}, $rd3 lpush src{t}, $rd3 set key2{t} in a pipeline
      
      The correct order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. lmove dst{t} src{t} right left
      5. set key2{t}
      
      The wrong order would be:
      1. set key1{t}
      2. lpush src{t}
      3. lmove src{t} dst{t} left right
      4. set key2{t}
      5. lmove dst{t} src{t} right left
      
      This PR adds corresponding test to cover it.
      
      * Add comment near while(listLength(server.ready_keys) != 0)
      9dc6f93e
  2. 15 Jun, 2023 1 commit
  3. 14 Jun, 2023 1 commit
  4. 13 Jun, 2023 1 commit
    • Binbin's avatar
      Fix XREADGROUP BLOCK stuck in endless loop (#12301) · e7129e43
      Binbin authored
      
      
      For the XREADGROUP BLOCK > scenario, there is an endless loop.
      Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command
      
      The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
      looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
      and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
      i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
      is problematic since the code keeps accessing the global list.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      e7129e43
  5. 12 Jun, 2023 1 commit
  6. 11 Jun, 2023 3 commits
    • YaacovHazan's avatar
      Reset command duration for rejected command. (#12247) · 0bfb6d55
      YaacovHazan authored
      In 7.2, After 971b177f
      
       we make sure (assert) that
      the duration has been recorded when resetting the client.
      
      This is not true for rejected commands.
      The use case I found is a blocking command that an ACL rule changed before
      it was unblocked, and while reprocessing it, the command rejected and triggered the assert.
      
      The PR reset the command duration inside rejectCommand / rejectCommandSds.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0bfb6d55
    • Chen Tianjie's avatar
      Allow bigger tolerance in eventloop duration test. (#12179) · 2d7d3911
      Chen Tianjie authored
      In #11963, some new tests about eventloop duration were added, which includes time measurement in TCL scripts. This has caused some unexpected CI failures, such as #12169 and #12177, due to slow test servers or some performance jittering.
      2d7d3911
    • Wen Hui's avatar
      Adding missing test cases for Addslot Command (#12288) · d412269f
      Wen Hui authored
      Added missing test case coverage for below scenarios:
      
      1. The command only works if all the specified slots are, from
        the point of view of the node receiving the command, currently
        not assigned. A node will refuse to take ownership for slots that
        already belong to some other node (including itself).
      2. The command fails if the same slot is specified multiple times.
      d412269f
  7. 28 May, 2023 3 commits
    • Oran Agra's avatar
      Optimize MSETNX to avoid double lookup (#11944) · 2764dc37
      Oran Agra authored
      This is a redo of #11594 which got reverted in #11940
      It improves performance by avoiding double lookup of the the key.
      2764dc37
    • Oran Agra's avatar
      Fix WAIT for clients being blocked in a module command (#12220) · 6117f288
      Oran Agra authored
      So far clients being blocked and unblocked by a module command would
      update the c->woff variable and so WAIT was ineffective and got released
      without waiting for the command actions to propagate.
      
      This seems to have existed since forever, but not for RM_BlockClientOnKeys.
      
      It is problematic though to know if the module did or didn't propagate
      anything in that command, so for now, instead of adding an API, we'll
      just update the woff to the latest offset when unblocking, this will
      cause the client to possibly wait excessively, but that's not that bad.
      6117f288
    • Wen Hui's avatar
      [BUG] Incorrect error msg for XREAD command (#12238) · 1a188e4e
      Wen Hui authored
      
      
      XREAD only supports a special ID of $ and XREADGROUP only supports ^.
      make sure not to suggest the wrong one when rerunning an error about unbalanced ID arguments
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1a188e4e
  8. 24 May, 2023 2 commits
    • judeng's avatar
      postpone the initialization of oject's lru&lfu until it is added to the db as... · d71478a8
      judeng authored
      postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
      
      This pr can get two performance benefits:
      1. Stop redundant initialization when most robj objects are created
      2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
      
      Another code optimization:
      deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
      robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
      d71478a8
    • Wen Hui's avatar
      Adding test case for hvals, hkeys, hexists against wrong type (#12198) · d6648899
      Wen Hui authored
      HVALS, HKEYS and HEXISTS commands wrong type test cases were not covered so added the test cases.
      d6648899
  9. 23 May, 2023 1 commit
  10. 18 May, 2023 2 commits
    • Binbin's avatar
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and... · 48757934
      Binbin authored
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and expand dict in advance (#12185)
      
      For zsets that will eventually be stored as the skiplist encoding (has a dict),
      we can convert it to skiplist ahead of time. This change checks the number
      of arguments in the ZADD command, and converts the data-structure
      if the number of new entries exceeds the listpack-max-entries configuration.
      This can cause us to over-allocate memory if there are duplicate entries in the
      input, which is unexpected.
      
      For ZRANGESTORE, we know the size of the zset, so we can expand
      the dict in advance, to avoid the temporary dict from being rehashed
      while it grows.
      
      Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
      48757934
    • Hanna Fadida's avatar
      Add BITFIELD_RO basic tests for non-repl use cases (#12187) · 37cf1984
      Hanna Fadida authored
      Current tests for BITFIELD_RO command are skipped in the external mode,
      and therefore reply-schemas-validator reports a coverage error.
      This PR adds basic tests to increase coverage.
      37cf1984
  11. 16 May, 2023 2 commits
    • Binbin's avatar
      Fix for set max entries edge case in setTypeCreate / setTypeMaybeConvert (#12183) · fd566f40
      Binbin authored
      In the judgment in setTypeCreate, we should judge size_hint <= max_entries.
      
      This results in the following inconsistencies:
      ```
      127.0.0.1:6379> config set set-max-intset-entries 5 set-max-listpack-entries 5
      OK
      
      127.0.0.1:6379> sadd intset_set1 1 2 3 4 5
      (integer) 5
      127.0.0.1:6379> object encoding intset_set1
      "hashtable"
      127.0.0.1:6379> sadd intset_set2 1 2 3 4
      (integer) 4
      127.0.0.1:6379> sadd intset_set2 5
      (integer) 1
      127.0.0.1:6379> object encoding intset_set2
      "intset"
      
      127.0.0.1:6379> sadd listpack_set1 a 1 2 3 4
      (integer) 5
      127.0.0.1:6379> object encoding listpack_set1
      "hashtable"
      127.0.0.1:6379> sadd listpack_set2 a 1 2 3
      (integer) 4
      127.0.0.1:6379> sadd listpack_set2 4
      (integer) 1
      127.0.0.1:6379> object encoding listpack_set2
      "listpack"
      ```
      
      This was introduced in #12019, added corresponding tests.
      fd566f40
    • Wen Hui's avatar
      Adding missing test case for smembers scard commands (#12148) · e4527288
      Wen Hui authored
      Minor missing test case addition.
      
      SMEMBERS SCARD against non set
      SMEMBERS SCARD against non existing key
      e4527288
  12. 14 May, 2023 2 commits
  13. 12 May, 2023 1 commit
    • Chen Tianjie's avatar
      Add basic eventloop latency measurement. (#11963) · 29ca8795
      Chen Tianjie authored
      The measured latency(duration) includes the list below, which can be shown by `INFO STATS`.
      ```
      eventloop_cycles  // ever increasing counter
      eventloop_duration_sum // cumulative duration of eventloop in microseconds
      eventloop_duration_cmd_sum  // cumulative duration of executing commands in microseconds
      instantaneous_eventloop_cycles_per_sec  // average eventloop count per second in recent 1.6s
      instantaneous_eventloop_duration_usec  // average single eventloop duration in recent 1.6s
      ```
      
      Also added some experimental metrics, which are shown only when `INFO DEBUG` is called.
      This section isn't included in the default INFO, or even in `INFO ALL` and the fields in this section
      can change in the future without considering backwards compatibility.
      ```
      eventloop_duration_aof_sum  // cumulative duration of writing AOF
      eventloop_duration_cron_sum  // cumulative duration cron jobs (serverCron, beforeSleep excluding IO and AOF)
      eventloop_cmd_per_cycle_max  // max number of commands executed in one eventloop
      eventloop_duration_max  // max duration of one eventloop
      ```
      
      All of these are being reset by CONFIG RESETSTAT
      29ca8795
  14. 09 May, 2023 1 commit
    • Binbin's avatar
      EXPIRE precision test more attempts and more tolerant (#12150) · 6ab2174d
      Binbin authored
      The test failed on MacOS:
      ```
      *** [err]: EXPIRE precision is now the millisecond in tests/unit/expire.tcl
      Expected 'somevalue {}' to equal or match '{} {}'
      ```
      
      `set a [r get x]`, even though we tried 10 times, sometimes we
      still get {}, this is a time-sensitive test.
      
      In this PR, we add the following changes:
      1. More attempts, change it from 10 to 30.
      2. More tolerant, change the `after 900` to `after 800`.
      
      In addition, we judging $a in advance and changing `after 1100`
      to `after 300`, this will save us some times.
      6ab2174d
  15. 08 May, 2023 1 commit
  16. 07 May, 2023 2 commits
  17. 04 May, 2023 2 commits
    • Binbin's avatar
      Pause cron to prevent premature shrinking in querybuf test (#12126) · e49c2a52
      Binbin authored
      Tests occasionally fail since #12000:
      ```
      *** [err]: query buffer resized correctly when not idle in tests/unit/querybuf.tcl
      Expected 0 > 32768 (context: type eval line 11 cmd {assert {$orig_test_client_qbuf > 32768}} proc ::test)
      
      *** [err]: query buffer resized correctly with fat argv in tests/unit/querybuf.tcl
      query buffer should not be resized when client idle time smaller than 2s
      ```
      
      The reason may be because we set hz to 100, querybuf shrinks before we count
      client_query_buffer. We avoid this problem by setting pause-cron to 1.
      e49c2a52
    • guybe7's avatar
      multi.tcl: reset readraw at the end of the test (#12123) · 857c09b0
      guybe7 authored
      1. reset the readraw mode after a test that uses it. undetected since the
        only test after that on the same server didn't read any replies.
      2. fix a cross slot issue that was undetected in cluster mode because
        readraw doesn't throw exceptions on errors.
      857c09b0
  18. 01 May, 2023 1 commit
  19. 25 Apr, 2023 1 commit
  20. 19 Apr, 2023 1 commit
  21. 18 Apr, 2023 2 commits
    • Binbin's avatar
      Tests: Do not save an RDB by default and add a SIGTERM default AOFRW test (#12064) · 20533cc1
      Binbin authored
      
      
      In order to speed up tests, avoid saving an RDB (mostly notable on shutdown),
      except for tests that explicitly test the RDB mechanism
      
      In addition, use `shutdown-on-sigterm force` to prevetn shutdown from failing
      in case the server is in the middle of the initial AOFRW
      
      Also a a test that checks that the `shutdown-on-sigterm default` is to refuse
      shutdown if there's an initial AOFRW
      Co-authored-by: default avatarGuy Benoish <guy.benoish@redislabs.com>
      20533cc1
    • Wen Hui's avatar
      Added getrange missing testcase (#12061) · d2db4aa7
      Wen Hui authored
      Minor test case addition.
      Currently GETRANGE command does not have the test case coverage for the scenarios:
      An error is returned when key exists but of different type
      Added missing test cases for getrange command.
      d2db4aa7
  22. 16 Apr, 2023 1 commit
    • judeng's avatar
      avoid incorrect shrinking of querybuf when client is reading a big argv (#12000) · e7f18432
      judeng authored
      this pr fix two wrongs:
      1. When client’s querybuf is pre-allocated for a fat argv, we need to update the
        querybuf_peak of the client immediately to completely avoid the unexpected
        shrinking of querybuf in the next clientCron (before data arrives to set the peak).
      2. the protocol's bulklen does not include `\r\n`, but the allocation and the data we
        read does. so in `clientsCronResizeQueryBuffer`, the `resize` or `querybuf_peak`
        should add these 2 bytes.
      
      the first bug is likely to hit us on large payloads over slow connections, in which case
      transferring the payload can take longer and a cron event will be triggered (specifically
      if there are not a lot of clients)
      e7f18432
  23. 13 Apr, 2023 2 commits
    • Wen Hui's avatar
      Adding missing test cases for substring (#12039) · 4375b01c
      Wen Hui authored
      There is are some missing test cases for SUBSTR command.
      These might already be covered by GETRANGE, but no harm in adding them since they're simple.
      
      Added 3 test case.
      
      * start > stop
      * start and stop both greater than string length
      * when no key is present.
      4375b01c
    • Wen Hui's avatar
      Adding missing test cases for linsert command (#12040) · bc82309c
      Wen Hui authored
      Currently LINSERT command does not have the test case coverage for following scenarios.
      1. When key does not exist, it is considered an empty list and no operation is performed.
      2. An error is returned when key exists but does not hold a list value.
      
      Added above two missing test cases for linsert command.
      bc82309c
  24. 12 Apr, 2023 2 commits
    • Binbin's avatar
      Add RM_ReplyWithErrorFormat that can support format (#11923) · bfec2d70
      Binbin authored
      * Add RM_ReplyWithErrorFormat that can support format
      
      Reply with the error create from a printf format and arguments.
      
      If the error code is already passed in the string 'fmt', the error
      code provided is used, otherwise the string "-ERR " for the generic
      error code is automatically added.
      
      The usage is, for example:
          RedisModule_ReplyWithErrorFormat(ctx, "An error: %s", "foo");
          RedisModule_ReplyWithErrorFormat(ctx, "-WRONGTYPE Wrong Type: %s", "foo");
      
      The function always returns REDISMODULE_OK.
      bfec2d70
    • Oran Agra's avatar
      Attempt to solve MacOS CI issues in GH Actions (#12013) · 997fa41e
      Oran Agra authored
      The MacOS CI in github actions often hangs without any logs. GH argues that
      it's due to resource utilization, either running out of disk space, memory, or CPU
      starvation, and thus the runner is terminated.
      
      This PR contains multiple attempts to resolve this:
      1. introducing pause_process instead of SIGSTOP, which waits for the process
        to stop before resuming the test, possibly resolving race conditions in some tests,
        this was a suspect since there was one test that could result in an infinite loop in that
       case, in practice this didn't help, but still a good idea to keep.
      2. disable the `save` config in many tests that don't need it, specifically ones that use
        heavy writes and could create large files.
      3. change the `populate` proc to use short pipeline rather than an infinite one.
      4. use `--clients 1` in the macos CI so that we don't risk running multiple resource
        demanding tests in parallel.
      5. enable `--verbose` to be repeated to elevate verbosity and print more info to stdout
        when a test or a server starts.
      997fa41e
  25. 11 Apr, 2023 1 commit
    • Binbin's avatar
      Add ZREMRANGEBYLEX basics tests to fix reply-schemas daily (#12021) · 45b8eea1
      Binbin authored
      We do have ZREMRANGEBYLEX tests, but it is a stress test
      marked with slow tag and then skipped in reply-schemas daily.
      
      In the past, we were able to succeed on a daily, i guess
      it was because there were some random command executions,
      such as corrupt-dump-fuzzy, which might call it.
      
      These test examples are taken from ZRANGEBYLEX basics test.
      45b8eea1
  26. 09 Apr, 2023 1 commit
    • Ozan Tezcan's avatar
      Add RM_RdbLoad and RM_RdbSave module API functions (#11852) · e55568ed
      Ozan Tezcan authored
      Add `RM_RdbLoad()` and `RM_RdbSave()` to load/save RDB files from the module API. 
      
      In our use case, we have our clustering implementation as a module. As part of this
      implementation, the module needs to trigger RDB save operation at specific points.
      Also, this module delivers RDB files to other nodes (not using Redis' replication).
      When a node receives an RDB file, it should be able to load the RDB. Currently,
      there is no module API to save/load RDB files. 
      
      
      This PR adds four new APIs:
      ```c
      RedisModuleRdbStream *RM_RdbStreamCreateFromFile(const char *filename);
      void RM_RdbStreamFree(RedisModuleRdbStream *stream);
      
      int RM_RdbLoad(RedisModuleCtx *ctx, RedisModuleRdbStream *stream, int flags);
      int RM_RdbSave(RedisModuleCtx *ctx, RedisModuleRdbStream *stream, int flags);
      ```
      
      The first step is to create a `RedisModuleRdbStream` object. This PR provides a function to
      create RedisModuleRdbStream from the filename. (You can load/save RDB with the filename).
      In the future, this API can be extended if needed: 
      e.g., `RM_RdbStreamCreateFromFd()`, `RM_RdbStreamCreateFromSocket()` to save/load
      RDB from an `fd` or a `socket`. 
      
      
      Usage:
      ```c
      /* Save RDB */
      RedisModuleRdbStream *stream = RedisModule_RdbStreamCreateFromFile("example.rdb");
      RedisModule_RdbSave(ctx, stream, 0);
      RedisModule_RdbStreamFree(stream);
      
      /* Load RDB */
      RedisModuleRdbStream *stream = RedisModule_RdbStreamCreateFromFile("example.rdb");
      RedisModule_RdbLoad(ctx, stream, 0);
      RedisModule_RdbStreamFree(stream);
      ```
      e55568ed