1. 24 May, 2023 5 commits
    • Oran Agra's avatar
      CI to validate commands.def is up to date (#12227) · c871db24
      Oran Agra authored
      and update recent SENTINEL CONFIG changes.
      c871db24
    • judeng's avatar
      postpone the initialization of oject's lru&lfu until it is added to the db as... · d71478a8
      judeng authored
      postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
      
      This pr can get two performance benefits:
      1. Stop redundant initialization when most robj objects are created
      2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
      
      Another code optimization:
      deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
      robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
      d71478a8
    • Wen Hui's avatar
      Adding test case for hvals, hkeys, hexists against wrong type (#12198) · d6648899
      Wen Hui authored
      HVALS, HKEYS and HEXISTS commands wrong type test cases were not covered so added the test cases.
      d6648899
    • Binbin's avatar
      Sync the new loglevel nothing to sentinel (#12223) · d0994c5b
      Binbin authored
      We add a new loglevel 'nothing' to disable logging in #12133.
      This PR syncs that config change to sentinel. Because in #11214
      we support modifying loglevel in runtime.
      
      Although I think sentinel doesn't need this nothing config,
      it's better to be consistent.
      d0994c5b
    • Binbin's avatar
      Add dummy CLUSTER SLAVES call tests to fix reply ci (#12224) · ec5721d6
      Binbin authored
      In #12166, we removed a call to CLUSTER SLAVES, which
      then caused reply-schemas ci to fail:
      ```
      WARNING! The following commands were not hit at all:
        cluster|slaves
        ERROR! at least one command was not hit by the tests
      ```
      
      Because we already have command output that cover CLUSTER REPLICAS
      elsewhere, here we simply add some dummy tests to fix the ci.
      ec5721d6
  2. 23 May, 2023 3 commits
    • Ping Xie's avatar
      Exclude aux fields from "cluster nodes" and "cluster replicas" output (#12166) · 4c74dd98
      Ping Xie authored
      This commit excludes aux fields from the output of the `cluster nodes` and `cluster replicas` command.
      We may decide to re-introduce them in some form or another in the future, but not in v7.2.
      4c74dd98
    • zhaozhao.zz's avatar
      add a new loglevel 'nothing' to disable logging (#12133) · 07ea2204
      zhaozhao.zz authored
      Users can record logs of different levels by setting the `loglevel`.
      However, sometimes there are many logs even at the warning level,
      which can affect the performance of Redis.
      
      For example, when a user accesses the tls-port using a non-encrypted link,
      Redis will log lots of "# Error accepting a client connection: ...".
      
      We can provide the ability to disable logging so that users can temporarily turn
      off logging and turn it back on after the problem is resolved.
      07ea2204
    • Shaya Potter's avatar
      Fix memory leak when RM_Call's RUN_AS_USER fails (#12158) · 71e6abe4
      Shaya Potter authored
      previously the argv wasn't freed so would leak.  not a common case, but should be handled.
      
      Solution: move RUN_AS_USER setup and error exit to the right place.
      this way, when we do `goto cleanup` (instead of return) it'll automatically do the right thing (including autoMemoryAdd)
      Removed the user argument from moduleAllocTempClient (reverted to the state before 6e993a5d
      
      )
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      71e6abe4
  3. 22 May, 2023 2 commits
    • Binbin's avatar
      Optimize HRANDFIELD and ZRANDMEMBER case 3 when listpack encoded (#12205) · 006ab26c
      Binbin authored
      Optimized HRANDFIELD and ZRANDMEMBER commands as in #8444,
      CASE 3 under listpack encoding. Boost optimization to CASE 2.5. 
      
      CASE 2.5 listpack only. Sampling unique elements, in non-random order.
      Listpack encoded hashes / zsets are meant to be relatively small, so
      HRANDFIELD_SUB_STRATEGY_MUL / ZRANDMEMBER_SUB_STRATEGY_MUL
      isn't necessary and we rather not make copies of the entries. Instead, we
      emit them directly to the output buffer.
      
      Simple benchmarks shows it provides some 400% improvement in HRANDFIELD
      and ZRANGESTORE both in CASE 3.
      
      Unrelated changes: remove useless setTypeRandomElements and fix a typo.
      006ab26c
    • binfeng-xin's avatar
      optimize spopwithcount propagation (#12082) · 38e284f1
      binfeng-xin authored
      
      
      A single SPOP with command with count argument resulted in many SPOP
      commands being propagated to the replica.
      This is inefficient because the key name is repeated many times, and is also
      being looked-up many times.
      also it results in high QPS metrics on the replica.
      To solve that, we flush batches of 1024 fields per SPOP command.
      Co-authored-by: default avatarzhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
      38e284f1
  4. 18 May, 2023 4 commits
    • Binbin's avatar
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and... · 48757934
      Binbin authored
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and expand dict in advance (#12185)
      
      For zsets that will eventually be stored as the skiplist encoding (has a dict),
      we can convert it to skiplist ahead of time. This change checks the number
      of arguments in the ZADD command, and converts the data-structure
      if the number of new entries exceeds the listpack-max-entries configuration.
      This can cause us to over-allocate memory if there are duplicate entries in the
      input, which is unexpected.
      
      For ZRANGESTORE, we know the size of the zset, so we can expand
      the dict in advance, to avoid the temporary dict from being rehashed
      while it grows.
      
      Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
      48757934
    • Hanna Fadida's avatar
      Add BITFIELD_RO basic tests for non-repl use cases (#12187) · 37cf1984
      Hanna Fadida authored
      Current tests for BITFIELD_RO command are skipped in the external mode,
      and therefore reply-schemas-validator reports a coverage error.
      This PR adds basic tests to increase coverage.
      37cf1984
    • Brennan's avatar
      Prevent repetitive backlog trimming (#12155) · 40e6131b
      Brennan authored
      When `replicationFeedSlaves()` serializes a command, it repeatedly calls
      `feedReplicationBuffer()` to feed it to the replication backlog piece by piece.
      It is unnecessary to call `incrementalTrimReplicationBacklog()` for every small
      amount of data added with `feedReplicationBuffer()` as the chance of the conditions
      being met for trimming are very low and these frequent calls add up to a notable
      performance cost. Instead, we will only attempt trimming when a new block is added
      to the replication backlog.
      
      Using redis-benchmark to saturate a local redis server indicated a performance
      improvement of around 3-3.5% for 100 byte SET commands with this change.
      40e6131b
    • YaacovHazan's avatar
      fix CMD_CALL_FROM_MODULE value (#12195) · 49845c24
      YaacovHazan authored
      CMD_CALL_FROM_MODULE overlapped CMD_CALL_REPROCESSING,
      changing its value to (1<<3)
      49845c24
  5. 17 May, 2023 1 commit
  6. 16 May, 2023 2 commits
    • Binbin's avatar
      Fix for set max entries edge case in setTypeCreate / setTypeMaybeConvert (#12183) · fd566f40
      Binbin authored
      In the judgment in setTypeCreate, we should judge size_hint <= max_entries.
      
      This results in the following inconsistencies:
      ```
      127.0.0.1:6379> config set set-max-intset-entries 5 set-max-listpack-entries 5
      OK
      
      127.0.0.1:6379> sadd intset_set1 1 2 3 4 5
      (integer) 5
      127.0.0.1:6379> object encoding intset_set1
      "hashtable"
      127.0.0.1:6379> sadd intset_set2 1 2 3 4
      (integer) 4
      127.0.0.1:6379> sadd intset_set2 5
      (integer) 1
      127.0.0.1:6379> object encoding intset_set2
      "intset"
      
      127.0.0.1:6379> sadd listpack_set1 a 1 2 3 4
      (integer) 5
      127.0.0.1:6379> object encoding listpack_set1
      "hashtable"
      127.0.0.1:6379> sadd listpack_set2 a 1 2 3
      (integer) 4
      127.0.0.1:6379> sadd listpack_set2 4
      (integer) 1
      127.0.0.1:6379> object encoding listpack_set2
      "listpack"
      ```
      
      This was introduced in #12019, added corresponding tests.
      fd566f40
    • Wen Hui's avatar
      Adding missing test case for smembers scard commands (#12148) · e4527288
      Wen Hui authored
      Minor missing test case addition.
      
      SMEMBERS SCARD against non set
      SMEMBERS SCARD against non existing key
      e4527288
  7. 14 May, 2023 2 commits
  8. 12 May, 2023 1 commit
    • Chen Tianjie's avatar
      Add basic eventloop latency measurement. (#11963) · 29ca8795
      Chen Tianjie authored
      The measured latency(duration) includes the list below, which can be shown by `INFO STATS`.
      ```
      eventloop_cycles  // ever increasing counter
      eventloop_duration_sum // cumulative duration of eventloop in microseconds
      eventloop_duration_cmd_sum  // cumulative duration of executing commands in microseconds
      instantaneous_eventloop_cycles_per_sec  // average eventloop count per second in recent 1.6s
      instantaneous_eventloop_duration_usec  // average single eventloop duration in recent 1.6s
      ```
      
      Also added some experimental metrics, which are shown only when `INFO DEBUG` is called.
      This section isn't included in the default INFO, or even in `INFO ALL` and the fields in this section
      can change in the future without considering backwards compatibility.
      ```
      eventloop_duration_aof_sum  // cumulative duration of writing AOF
      eventloop_duration_cron_sum  // cumulative duration cron jobs (serverCron, beforeSleep excluding IO and AOF)
      eventloop_cmd_per_cycle_max  // max number of commands executed in one eventloop
      eventloop_duration_max  // max duration of one eventloop
      ```
      
      All of these are being reset by CONFIG RESETSTAT
      29ca8795
  9. 11 May, 2023 2 commits
  10. 10 May, 2023 1 commit
  11. 09 May, 2023 2 commits
    • Leibale Eidelman's avatar
      fix `GEORADIUS[BYMEMBER]` `STORE` & `STOREDIST` args spec (#12151) · e04ebdb8
      Leibale Eidelman authored
      in GEO commands, `STORE` and `STOREDIST` are mutually exclusive.
      use `oneof` block to contain them
      e04ebdb8
    • Binbin's avatar
      EXPIRE precision test more attempts and more tolerant (#12150) · 6ab2174d
      Binbin authored
      The test failed on MacOS:
      ```
      *** [err]: EXPIRE precision is now the millisecond in tests/unit/expire.tcl
      Expected 'somevalue {}' to equal or match '{} {}'
      ```
      
      `set a [r get x]`, even though we tried 10 times, sometimes we
      still get {}, this is a time-sensitive test.
      
      In this PR, we add the following changes:
      1. More attempts, change it from 10 to 30.
      2. More tolerant, change the `after 900` to `after 800`.
      
      In addition, we judging $a in advance and changing `after 1100`
      to `after 300`, this will save us some times.
      6ab2174d
  12. 08 May, 2023 5 commits
    • tison's avatar
      Correct zrangeGenericCommand comment (#12136) · 9b53cb28
      tison authored
      Fix incorrect documentation about the arguments of ZRANGE commands
      9b53cb28
    • cui fliter's avatar
      Remove several instances of duplicate "the" in comments (#12144) · 03d50e0c
      cui fliter authored
      Remove several instances of duplicate "the" in comments
      03d50e0c
    • Madelyn Olson's avatar
      Minor performance improvement to SADD and HSET (#12019) · a129a601
      Madelyn Olson authored
      For sets and hashes that will eventually be stored as the hash encoding, it's much faster to immediately convert them to their hash encoding and then perform the insertions since it avoids the O(N) search and frequent reallocations. This change checks the number of arguments in the incoming command, and converts the data-structure if the number of new entries exceeds the listpack-max-entries configuration. This can cause us to over-allocate memory if their are duplicate entries in the input, which is unexpected.
      
      unstable
      
      Summary:
        throughput summary: 805.54 requests per second
        latency summary (msec):
                avg       min       p50       p95       p99       max
             61.908    25.680    68.351    73.279    75.967    79.295
      hset-improvement
      
      Summary:
        throughput summary: 4701.46 requests per second
        latency summary (msec):
                avg       min       p50       p95       p99       max
             10.546     0.832    11.959    12.471    13.119    14.967
      a129a601
    • Oran Agra's avatar
      Merge pull request #12141 from oranagra/fix_32bit_ci · 51af17e3
      Oran Agra authored
      Fix defrag CI for 32bit after merge of jemalloc 5.3
      51af17e3
    • Oran Agra's avatar
      Fix defrag CI for 32bit after merge of jemalloc 5.3 · cded14f3
      Oran Agra authored
      The new mallctl seems to set the output sz when EINVAL occors.
      that messes up the retry mechanism that does /2 on each iteration.
      cded14f3
  13. 07 May, 2023 3 commits
  14. 06 May, 2023 1 commit
    • zhaozhao.zz's avatar
      Free backlog only if rsi is invalid when master reboot (#12088) · b0dd7b32
      zhaozhao.zz authored
      When master reboot from RDB, if rsi in RDB is valid we should not free replication backlog, even if master_repl_offset or repl-offset is 0.
      
      Since if master doesn't send any data to replicas master_repl_offset is 0, it's a valid number.
      
      A clear example:
      
      1. start a master and apply some write commands, the master's master_repl_offset is 0 since it has no replicas.
      2. stop write commands on master, and start another instance and replicaof the master, trigger an FULLRESYNC
      3. the master's master_repl_offset is still 0 (set a large number for repl-ping-replica-period), do BGSAVE and restart the master
      4. master load master_repl_offset from RDB's rsi and it's still 0, and we should make sure replica can partially resync with master.
      b0dd7b32
  15. 04 May, 2023 2 commits
    • Binbin's avatar
      Pause cron to prevent premature shrinking in querybuf test (#12126) · e49c2a52
      Binbin authored
      Tests occasionally fail since #12000:
      ```
      *** [err]: query buffer resized correctly when not idle in tests/unit/querybuf.tcl
      Expected 0 > 32768 (context: type eval line 11 cmd {assert {$orig_test_client_qbuf > 32768}} proc ::test)
      
      *** [err]: query buffer resized correctly with fat argv in tests/unit/querybuf.tcl
      query buffer should not be resized when client idle time smaller than 2s
      ```
      
      The reason may be because we set hz to 100, querybuf shrinks before we count
      client_query_buffer. We avoid this problem by setting pause-cron to 1.
      e49c2a52
    • guybe7's avatar
      multi.tcl: reset readraw at the end of the test (#12123) · 857c09b0
      guybe7 authored
      1. reset the readraw mode after a test that uses it. undetected since the
        only test after that on the same server didn't read any replies.
      2. fix a cross slot issue that was undetected in cluster mode because
        readraw doesn't throw exceptions on errors.
      857c09b0
  16. 03 May, 2023 2 commits
    • Madelyn Olson's avatar
      Fix name of module API in documentation (#12119) · fbbedcf5
      Madelyn Olson authored
      API incorrectly uses ExecutionUnit instead of Notification.
      fbbedcf5
    • Madelyn Olson's avatar
      Remove prototypes with empty declarations (#12020) · 5e3be1be
      Madelyn Olson authored
      Technically declaring a prototype with an empty declaration has been deprecated since the early days of C, but we never got a warning for it. C2x will apparently be introducing a breaking change if you are using this type of declarator, so Clang 15 has started issuing a warning with -pedantic. Although not apparently a problem for any of the compiler we build on, if feels like the right thing is to properly adhere to the C standard and use (void).
      5e3be1be
  17. 02 May, 2023 1 commit
    • Hassaan Khan's avatar
      [redis-benchmark] Adding --seed option to seed the RNG (#11945) · 8163e816
      Hassaan Khan authored
      Adds ability to set the random seed so that more consistent repeatable benchmarks are possible.
      
      Example usage:
      
      Adding 2 hash items
      ```
      src/redis-benchmark -r 100 -n 2 --seed 250 hset myhash:__rand_int__ age __rand_int__
      ```
      
      Monitor:
      
      1st benchmark invocation:
      ```
      1679332814.824357 [0 127.0.0.1:36686] "hset" "myhash:000000000022" "age" "000000000069"
      1679332814.824404 [0 127.0.0.1:36690] "hset" "myhash:000000000007" "age" "000000000043"
      ```
      
      2nd benchmark invocation:
      ```
      1679332814.824357 [0 127.0.0.1:36686] "hset" "myhash:000000000022" "age" "000000000069"
      1679332814.824404 [0 127.0.0.1:36690] "hset" "myhash:000000000007" "age" "000000000043"
      ```
      8163e816
  18. 01 May, 2023 1 commit