1. 21 Mar, 2022 2 commits
  2. 20 Mar, 2022 1 commit
    • 郭伟光's avatar
      unblockClient: avoid to reset client when the client was shutdown-blocked (#10440) · fae5b1a1
      郭伟光 authored
      fix #10439. see https://github.com/redis/redis/pull/9872
      When executing SHUTDOWN we pause the client so we can un-pause it
      if the shutdown fails.
      this could happen during the timeout, if the shutdown is aborted, but could
      also happen from withing the initial `call()` to shutdown, if the rdb save fails.
      in that case when we return to `call()`, we'll crash if `c->cmd` has been set to NULL.
      
      The call stack is:
      ```
      unblockClient(c)
      replyToClientsBlockedOnShutdown()
      cancelShutdown()
      finishShutdown()
      prepareForShutdown()
      shutdownCommand()
      ```
      
      what's special about SHUTDOWN in that respect is that it can be paused,
      and then un-paused before the original `call()` returns.
      tests where added for both failed shutdown, and a followup successful one.
      fae5b1a1
  3. 18 Mar, 2022 1 commit
    • sundb's avatar
      Restore ::singledb after cluster test (#10441) · b9656adb
      sundb authored
      When ::singledb is 0, we will use db 9 for the test db.
      Since ::singledb is set to 1 in the cluster-related tests, but not restored, some subsequent
      tests associated with db 9 will fail.
      b9656adb
  4. 17 Mar, 2022 1 commit
  5. 16 Mar, 2022 5 commits
    • Madelyn Olson's avatar
    • Viktor Söderqvist's avatar
      Fix redis-cli CLUSTER SETSLOT race conditions (#10381) · 69017fa2
      Viktor Söderqvist authored
      After migrating a slot, send CLUSTER SETSLOT NODE to the destination
      node first to make sure the slot isn't left without an owner in case
      the destination node crashes before it is set as new owner.
      
      When informing the source node, it can happen that the destination
      node has already informed it and if the source node has lost its
      last slot, it has already turned itself into a replica. Redis-cli
      should ignore this error in this case.
      69017fa2
    • Binbin's avatar
      Fix module redact test for valgrind (#10432) · 61b7e591
      Binbin authored
      The new module redact test will fail with valgrind:
      ```
      [err]: modules can redact arguments in tests/unit/moduleapi/auth.tcl
      Expected 'slowlog reset' to be equal to 'auth.redact 1 (redacted) 3 (redacted)' (context: type eval line 12 cmd {assert_equal {slowlog reset} [lindex [lindex [r slowlog get] 2] 3]} proc ::test)
      ```
      
      The reason is that with `slowlog-log-slower-than 10000`,
      `slowlog get` will have a chance to exceed 10ms.
      
      Made two changes to avoid failure:
      1. change `slowlog-log-slower-than` from 10000 to -1, distable it.
      2. assert to use the previous execution result.
      
      In theory, the second one can actually be left unchanged, but i
      think it will be better if it is changed.
      61b7e591
    • Harkrishn Patro's avatar
      Add new cluster shards command (#10293) · 45ccae89
      Harkrishn Patro authored
      
      
      Implement a new cluster shards command, which provides a flexible and extensible API for topology discovery.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      45ccae89
    • Madelyn Olson's avatar
      Add module API for redacting command arguments (#10425) · 416c9ac2
      Madelyn Olson authored
      Add module API for redacting client commands
      416c9ac2
  6. 15 Mar, 2022 3 commits
    • Wen Hui's avatar
      Sentinel: update command json files (#10374) · c30de707
      Wen Hui authored
      c30de707
    • ranshid's avatar
      make sort/ro commands validate external keys access patterns (#10106) (#10340) · 1078e30c
      ranshid authored
      
      
      Currently the sort and sort_ro can access external keys via `GET` and `BY`
      in order to make sure the user cannot violate the authorization ACL
      rules, the decision is to reject external keys access patterns unless ACL allows
      SORT full access to all keys.
      I.e. for backwards compatibility, SORT with GET/BY keeps working, but
      if ACL has restrictions to certain keys, these features get permission denied.
      
      ### Implemented solution
      We have discussed several potential solutions and decided to only allow the GET and BY
      arguments when the user has all key permissions with the SORT command. The reasons
      being that SORT with GET or BY is problematic anyway, for instance it is not supported in
      cluster mode since it doesn't declare keys, and we're not sure the combination of that feature
      with ACL key restriction is really required.
      **HOWEVER** If in the fullness of time we will identify a real need for fine grain access
      support for SORT, we would implement the complete solution which is the alternative
      described below.
      
      ### Alternative (Completion solution):
      Check sort ACL rules after executing it and before committing output (either via store or
      to COB). it would require making several changes to the sort command itself. and would
      potentially cause performance degradation since we will have to collect all the get keys
      instead of just applying them to a temp array and then scan the access keys against the
      ACL selectors. This solution can include an optimization to avoid the overheads of collecting
      the key names, in case the ACL rules grant SORT full key-access, or if the ACL key pattern
      literal matches the one used in GET/BY. It would also mean that authorization would be
      O(nlogn) since we will have to complete most of the command execution before we can
      perform verification
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1078e30c
    • yoav-steinberg's avatar
      Optimization: remove `updateClientMemUsage` from i/o threads. (#10401) · cf6dcb7b
      yoav-steinberg authored
      In a benchmark we noticed we spend a relatively long time updating the client
      memory usage leading to performance degradation.
      Before #8687 this was performed in the client's cron and didn't affect performance.
      But since introducing client eviction we need to perform this after filling the input
      buffers and after processing commands. This also lead me to write this code to be
      thread safe and perform it in the i/o threads.
      
      It turns out that the main performance issue here is related to atomic operations
      being performed while updating the total clients memory usage stats used for client
      eviction (`server.stat_clients_type_memory[]`). This update needed to be atomic
      because `updateClientMemUsage()` was called from the IO threads.
      
      In this commit I make sure to call `updateClientMemUsage()` only from the main thread.
      In case of threaded IO I call it for each client during the "fan-in" phase of the read/write
      operation. This also means I could chuck the `updateClientMemUsageBucket()` function
      which was called during this phase and embed it into `updateClientMemUsage()`.
      
      Profiling shows this makes `updateClientMemUsage()` (on my x86_64 linux) roughly x4 faster.
      cf6dcb7b
  7. 14 Mar, 2022 2 commits
  8. 13 Mar, 2022 3 commits
  9. 10 Mar, 2022 4 commits
    • Binbin's avatar
      Initialize help when using redis-cli help or redis-cli ? (#10382) · 1797330e
      Binbin authored
      The following usage will output an empty newline:
      ```
      > redis-cli help set
      empty line
      ```
      
      The reason is that in interactive mode, we have called
      `cliInitHelp`, which initializes help.
      
      When using `redis-cli help xxx` or `redis-cli help ? xxx`,
      we can't match the command due to empty `helpEntries`,
      so we output an empty newline.
      
      In this commit, we will call `cliInitHelp` to init the help.
      Note that in this case, we need to call `cliInitHelp` (COMMAND DOCS)
      every time, which i think is acceptable.
      
      So now the output will look like:
      ```
      [redis]# src/redis-cli help get
      
        GET key
        summary: Get the value of a key
        since: 1.0.0
        group: string
      
      [redis]#
      ```
      
      Fixes #10378
      
      This PR also fix a redis-cli crash when using `--ldb --eval`:
      ```
      [root]# src/redis-cli --ldb --eval test.lua test 1
      Lua debugging session started, please use:
      quit    -- End the session.
      restart -- Restart the script in debug mode again.
      help    -- Show Lua script debugging commands.
      
      * Stopped at 1, stop reason = step over
      -> 1   local num = redis.call('GET', KEYS[1]);
      redis-cli: redis-cli.c:718: cliCountCommands: Assertion
      `commandTable->element[i]->type == 1' failed.
      Aborted
      ```
      Because in ldb mode, `COMMAND DOCS` or `COMMAND` will
      return an array, only with one element, and the type
      is `REDIS_REPLY_STATUS`, the result is `<error> Unknown
      Redis Lua debugger command or wrong number of arguments`.
      
      So if we are in the ldb mode, and init the Redis HELP, we
      will get the wrong response and crash the redis-cli.
      In ldb mode we don't initialize HELP, help is only initialized
      after the lua debugging session ends.
      
      It was broken in #10043
      1797330e
    • ranshid's avatar
      ACL DRYRUN does not validate the verified command args. (#10405) · 11b071a2
      ranshid authored
      As a result we segfault when parsing and matching the command keys.
      11b071a2
    • zhugezy's avatar
      set "disable-thp" config immutable (#10409) · a26cab9d
      zhugezy authored
      It's confusing for this config to be modifiable since it only takes effect on startup
      a26cab9d
    • rangerzhang's avatar
      Fix outdated comments on updateSlavesWaitingBgsave (#10394) · 4e012dae
      rangerzhang authored
      * fix-replication-comments
      
      The described capacity
       `and to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress`
      was moved to `checkChildrenDone()`  named by `replicationStartPendingFork`
      
      But the comment was not changed, may misleading others.
      
      * remove-misleading-comments
      
      The described capacity
       `to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress` 
      and 
      `or when the replication RDB transfer strategy is modified from disk to socket or the other way around` 
      were not correct now.
      4e012dae
  10. 09 Mar, 2022 4 commits
  11. 08 Mar, 2022 6 commits
    • Ronald Petty's avatar
      Update redis.conf (#10396) · b104f3ca
      Ronald Petty authored
      Typo in conf file comment.
      b104f3ca
    • guybe7's avatar
      XREADGROUP: Unblock client if stream is deleted (#10306) · 2a295408
      guybe7 authored
      Deleting a stream while a client is blocked XREADGROUP should unblock the client.
      
      The idea is that if a client is blocked via XREADGROUP is different from
      any other blocking type in the sense that it depends on the existence of both
      the key and the group. Even if the key is deleted and then revived with XADD
      it won't help any clients blocked on XREADGROUP because the group no longer
      exist, so they would fail with -NOGROUP anyway.
      The conclusion is that it's better to unblock these clients (with error) upon
      the deletion of the key, rather than waiting for the first XADD. 
      
      Other changes:
      1. Slightly optimize all `serveClientsBlockedOn*` functions by checking `server.blocked_clients_by_type`
      2. All `serveClientsBlockedOn*` functions now use a list iterator rather than looking at `listFirst`, relying
        on `unblockClient` to delete the head of the list. Before this commit, only `serveClientsBlockedOnStreams`
        used to work like that.
      3. bugfix: CLIENT UNBLOCK ERROR should work even if the command doesn't have a timeout_callback
        (only relevant to module commands)
      2a295408
    • zhaozhao.zz's avatar
      script should not allow may-replicate commands when client pause write (#10364) · 728e6252
      zhaozhao.zz authored
      In some special commands like eval_ro / fcall_ro we allow no-writes commands.
      But may-replicate commands are no-writes too, that leads crash when client pause write:
      728e6252
    • Oran Agra's avatar
      dismiss COW of client output buffer now that it's dynamic (#10371) · b3fe4f31
      Oran Agra authored
      since #9822, the static reply buffer is no longer part of the client structure, so we need to dismiss it.
      b3fe4f31
    • zhugezy's avatar
      remove a piece of redundant comment (#10392) · 4f19b4d0
      zhugezy authored
      introduced in #10147 since we blocked the first-arg mechanism on subcommands
      4f19b4d0
    • Yossi Gottlieb's avatar
      Fix redis-benchmark --cluster with IPv6. (#10393) · 38052fd7
      Yossi Gottlieb authored
      Currently, CLUSTER NODES is parsed and was not done correctly for IPv6
      addresses.
      38052fd7
  12. 07 Mar, 2022 3 commits
  13. 06 Mar, 2022 1 commit
  14. 05 Mar, 2022 1 commit
    • Yuta Hongo's avatar
      redis-cli: Better --json Unicode support and --quoted-json (#10286) · e3ef73dc
      Yuta Hongo authored
      Normally, `redis-cli` escapes non-printable data received from Redis, using a custom scheme (which is also used to handle quoted input). When using `--json` this is not desired as it is not compatible with RFC 7159, which specifies JSON strings are assumed to be Unicode and how they should be escaped.
      
      This commit changes `--json` to follow RFC 7159, which means that properly encoded Unicode strings in Redis will result with a valid Unicode JSON.
      
      However, this introduces a new problem with `--json` and data that is not valid Unicode (e.g., random binary data, text that follows other encoding, etc.). To address this, we add `--quoted-json` which produces JSON strings that follow the original redis-cli quoting scheme.
      
      For example, a value that consists of only null (0x00) bytes will show up as:
      * `"\u0000\u0000\u0000"` when using `--json`
      * `"\\x00\\x00\\x00"` when using `--quoted-json`
      e3ef73dc
  15. 03 Mar, 2022 1 commit
  16. 02 Mar, 2022 1 commit
    • Henry's avatar
      A faster and more robust code of zslRandomLevel using RAND_MAX (#5539) · feb032fd
      Henry authored
      1. since ZSKIPLIST_P is float, using it directly inside the condition used to causes floating point code to be used (gcc/x86)
      2. In some operating system(eg.Windows), the largest value returned from random() is 0x7FFF(15bit), so after bitwise AND with 0xFFFF, the probability of the less operation returning true in the while loop's condition is no more equal to ZSKIPLIST_P.
      3. In case some library has random() returning int in range [0~ZSKIPLIST_P*65535], the while loop will be an infinite loop.
      4. on Linux where RAND_MAX is higher than 0xFFFF, this change actually improves precision (despite not matching the result against a float value)
      feb032fd
  17. 01 Mar, 2022 1 commit
    • ranshid's avatar
      Introduce debug command to disable reply buffer resizing (#10360) · 9b15dd28
      ranshid authored
      In order to resolve some flaky tests which hard rely on examine memory footprint.
      we introduce the following fixes:
      
      # Fix in client-eviction test - by @yoav-steinberg 
      Sometime the libc allocator can use different size client struct allocations.
      this may cause unexpected memory calculations to fail the test.
      
      # Introduce new DEBUG command for disabling reply buffer resizing
      In order to eliminate reply buffer resizing during specific tests.
      we introduced the ability to disable (and enable) the resizing cron job
      
      Co-authored-by: yoav-steinberg yoav@redislabs.com
      9b15dd28