1. 26 Jul, 2022 1 commit
    • Viktor Söderqvist's avatar
      Gossip forgotten nodes on `CLUSTER FORGET` (#10869) · 5032de50
      Viktor Söderqvist authored
      Gossip the cluster node blacklist in ping and pong messages.
      This means that CLUSTER FORGET doesn't need to be sent to all nodes in a cluster.
      It can be sent to one or more nodes and then be propagated to the rest of them.
      
      For each blacklisted node, its node id and its remaining blacklist TTL is gossiped in a
      cluster bus ping extension (introduced in #9530).
      5032de50
  2. 19 Jul, 2022 1 commit
    • Binbin's avatar
      Fix timing issue in cluster test (#11008) · 5ce64ab0
      Binbin authored
      A timing issue like this was reported in freebsd daily CI:
      ```
      *** [err]: Sanity test push cmd after resharding in tests/unit/cluster/cli.tcl
      Expected 'CLUSTERDOWN The cluster is down' to match '*MOVED*'
      ```
      
      We additionally wait for each node to reach a consensus on the cluster
      state in wait_for_condition to avoid the cluster down error.
      
      The fix just like #10495, quoting madolson's comment:
      Cluster check just verifies the the config state is self-consistent,
      waiting for cluster_state to be okay is an independent check that all
      the nodes actually believe each other are healthy.
      
      At the same time i noticed that unit/moduleapi/cluster.tcl has an exact
      same test, may have the same problem, also modified it.
      5ce64ab0
  3. 18 Jul, 2022 2 commits
    • Oran Agra's avatar
      Fix heap overflow corruption in XAUTOCLAIM (CVE-2022-31144) (#11002) · 2825b605
      Oran Agra authored
      The temporary array for deleted entries reply of XAUTOCLAIM was
      insufficient, but also in fact the COUNT argument should be used to
      control the size of the reply, so instead of terminating the loop by
      only counting the claimed entries, we'll count deleted entries as well.
      
      Fix #10968
      Addresses CVE-2022-31144
      2825b605
    • ranshid's avatar
      Avoid using unsafe C functions (#10932) · eacca729
      ranshid authored
      replace use of:
      sprintf --> snprintf
      strcpy/strncpy  --> redis_strlcpy
      strcat/strncat  --> redis_strlcat
      
      **why are we making this change?**
      Much of the code uses some unsafe variants or deprecated buffer handling
      functions.
      While most cases are probably not presenting any issue on the known path
      programming errors and unterminated strings might lead to potential
      buffer overflows which are not covered by tests.
      
      **As part of this PR we change**
      1. added implementation for redis_strlcpy and redis_strlcat based on the strl implementation: https://linux.die.net/man/3/strl
      2. change all occurrences of use of sprintf with use of snprintf
      3. change occurrences of use of  strcpy/strncpy with redis_strlcpy
      4. change occurrences of use of strcat/strncat with redis_strlcat
      5. change the behavior of ll2string/ull2string/ld2string so that it will always place null
        termination ('\0') on the output buffer in the first index. this was done in order to make
        the use of these functions more safe in cases were the user will not check the output
        returned by them (for example in rdbRemoveTempFile)
      6. we added a compiler directive to issue a deprecation error in case a use of
        sprintf/strcpy/strcat is found during compilation which will result in error during compile time.
        However keep in mind that since the deprecation attribute is not supported on all compilers,
        this is expected to fail during push workflows.
      
      
      **NOTE:** while this is only an initial milestone. We might also consider
      using the *_s implementation provided by the C11 Extensions (however not
      yet widly supported). I would also suggest to start
      looking at static code analyzers to track unsafe use cases.
      For example LLVM clang checker supports security.insecureAPI.DeprecatedOrUnsafeBufferHandling
      which can help locate unsafe function usage.
      https://clang.llvm.org/docs/analyzer/checkers.html#security-insecureapi-deprecatedorunsafebufferhandling-c
      The main reason not to onboard it at this stage is that the alternative
      excepted by clang is to use the C11 extensions which are not always
      supported by stdlib.
      eacca729
  4. 17 Jul, 2022 2 commits
  5. 13 Jul, 2022 1 commit
    • Oran Agra's avatar
      Avoid valgrind fishy value warning on corrupt restore payloads (#10937) · 599e59eb
      Oran Agra authored
      The corrupt dump fuzzer uncovered a valgrind warning saying:
      ```
      ==76370== Argument 'size' of function malloc has a fishy (possibly negative) value: -3744781444216323815
      ```
      This allocation would have failed (returning NULL) and being handled properly by redis (even before this change), but we also want to silence the valgrind warnings (which are checking that casting to ssize_t produces a non-negative value).
      
      The solution i opted for is to explicitly fail these allocations (returning NULL), before even reaching `malloc` (which would have failed and return NULL too).
      
      The implication is that we will not be able to support a single allocation of more than 2GB on a 32bit system (which i don't think is a realistic scenario).
      i.e. i do think we could be facing cases were redis consumes more than 2gb on a 32bit system, but not in a single allocation.
      
      The byproduct of this, is that i dropped the overflow assertions, since these will now lead to the same OOM panic we have for failed allocations.
      599e59eb
  6. 12 Jul, 2022 1 commit
  7. 11 Jul, 2022 3 commits
    • Binbin's avatar
      Trying to fix cluster test (#10963) · 693acc01
      Binbin authored
      #10942 break the new test added in #10449
      ```
      Testing unit: 29-slot-migration-response.tcl
      Cluster Join and auto-discovery test: FAILED: Cluster failed to join into a full mesh.
      ```
      
      It looks like we need to wait for the cluster in 28 to become stable.
      693acc01
    • Binbin's avatar
      Add cluster-port support to redis-cli --cluster (#10344) · 35e8ae3e
      Binbin authored
      
      
      In #9389, we add a new `cluster-port` config and make cluster bus port configurable,
      and currently redis-cli --cluster create/add-node doesn't support with a configurable `cluster-port` instance.
      Because redis-cli uses the old way (port + 10000) to send the `CLUSTER MEET` command.
      
      Now we add this support on redis-cli `--cluster`, note we don't need to explicitly pass in the
      `cluster-port` parameter, we can get the real `cluster-port` of the node in `clusterManagerNodeLoadInfo`,
      so the `--cluster create` and `--cluster add-node` interfaces have not changed.
      
      We will use the `cluster-port` when we are doing `CLUSTER MEET`, also note that `CLUSTER MEET` bus-port
      parameter was added in 4.0, so if the bus_port (the one in redis-cli) is 0, or equal (port + 10000),
      we just call `CLUSTER MEET` with 2 arguments, using the old form.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      35e8ae3e
    • Madelyn Olson's avatar
      Fix crash during handshake and cluster shards call (#10942) · e6a1b2ea
      Madelyn Olson authored
      * Fix an engine crash when there are nodes in handshaking and a user calls cluster shards
      e6a1b2ea
  8. 04 Jul, 2022 2 commits
  9. 03 Jul, 2022 1 commit
  10. 30 Jun, 2022 1 commit
    • Binbin's avatar
      Add SENTINEL command flag to CLIENT/COMMANDS subcommands (#10904) · 35e836c2
      Binbin authored
      This was harmless because we marked the parent command
      with SENTINEL flag. So the populateCommandTable was ok.
      And we also don't show the flag (SENTINEL and ONLY-SENTNEL)
      in COMMAND INFO.
      
      In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL
      flags check when populating the sub-commands.
      so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
      35e836c2
  11. 29 Jun, 2022 1 commit
  12. 28 Jun, 2022 1 commit
    • jonnyomerredis's avatar
      Add sharded pubsub keychannel count to client info (#10895) · 35c2ee87
      jonnyomerredis authored
      When calling CLIENT INFO/LIST, and in various debug prints, Redis is printing
      the number of pubsub channels / patterns the client is subscribed to.
      With the addition of sharded pubsub, it would be useful to print the number of
      keychannels the client is subscribed to as well.
      35c2ee87
  13. 27 Jun, 2022 1 commit
    • Viktor Söderqvist's avatar
      Add missing REDISMODULE_CLIENTINFO_INITIALIZER (#10885) · 6af02100
      Viktor Söderqvist authored
      The module API docs mentions this macro, but it was not defined (so no one could have used it).
      
      Instead of adding it as is, we decided to add a _V1 macro, so that if / when we some day extend this struct,
      modules that use this API and don't need the extra fields, will still use the old version
      and still be compatible with older redis version (despite being compiled with newer redismodule.h)
      6af02100
  14. 26 Jun, 2022 3 commits
    • RinChanNOW!'s avatar
      Support conversion between `RedisModuleString` and `unsigned long long` (#10889) · 28546373
      RinChanNOW! authored
      
      
      Since the ranges of `unsigned long long` and `long long` are different, we cannot read an
      `unsigned long long` integer from a `RedisModuleString` by `RedisModule_StringToLongLong` . 
      
      So I added two new Redis Module APIs to support the conversion between these two types:
      * `RedisModule_StringToULongLong`
      * `RedisModule_CreateStringFromULongLong`
      Signed-off-by: default avatarRinChanNOWWW <hzy427@gmail.com>
      28546373
    • Binbin's avatar
      redis-server command line arguments allow passing config name and value in the same arg (#10866) · d443e312
      Binbin authored
      This commit has two topics.
      
      ## Passing config name and value in the same arg
      In #10660 (Redis 7.0.1), when we supported the config values that can start with `--` prefix (one of the two topics of that PR),
      we broke another pattern: `redis-server redis.config "name value"`, passing both config name
      and it's value in the same arg, see #10865
      
      This wasn't a intended change (i.e we didn't realize this pattern used to work).
      Although this is a wrong usage, we still like to fix it.
      
      Now we support something like:
      ```
      src/redis-server redis.conf "--maxmemory '700mb'" "--maxmemory-policy volatile-lru" --proc-title-template --my--title--template --loglevel verbose
      ```
      
      ## Changes around --save
      Also in this PR, we undo the breaking change we made in #10660 on purpose.
      1. `redis-server redis.conf --save --loglevel verbose` (missing `save` argument before anotehr argument).
          In 7.0.1, it was throwing an wrong arg error.
          Now it will work and reset the save, similar to how it used to be in 7.0.0 and 6.2.x.
      3. `redis-server redis.conf --loglevel verbose --save` (missing `save` argument as last argument).
          In 6.2, it did not reset the save, which was a bug (inconsistent with the previous bullet).
          Now we will make it work and reset the save as well (a bug fix).
      d443e312
    • Viktor Söderqvist's avatar
      Add RM_SetClientNameById and RM_GetClientNameById (#10839) · 6272ca60
      Viktor Söderqvist authored
      Adding Module APIs to let the module read and set the client name of an arbitrary connection.
      6272ca60
  15. 23 Jun, 2022 1 commit
    • judeng's avatar
      fix benchmark failure in daily test with TLS (#10896) · d2405b9b
      judeng authored
      The new test added in #10891 can fail with a different error.
      see comment in networking.c saying
      ```c
              /* That's a best effort error message, don't check write errors.
               * Note that for TLS connections, no handshake was done yet so nothing
               * is written and the connection will just drop. */
      ```
      d2405b9b
  16. 22 Jun, 2022 1 commit
    • judeng's avatar
      fix redis-benchmark's bug: check if clients are created successfully in idle mode (#10891) · 49876158
      judeng authored
      my maxclients config:
      ```
      redis-cli config get maxclients
      1) "maxclients"
      2) "4064"
      ```
      
      Before this bug was fixed, creating 4065 clients appeared to be successful, but only 4064 were actually created```
      ```
      ./redis-benchmark -c 4065 -I
      Creating 4065 idle connections and waiting forever (Ctrl+C when done)
      cients: 4065
      ```
      
      now :
      ```
      ./redis-benchmark -c 4065 -I
      Creating 4065 idle connections and waiting forever (Ctrl+C when done)
      Error from server: ERR max number of clients reached
      
      ./redis-benchmark -c 4064 -I
      Creating 4064 idle connections and waiting forever (Ctrl+C when done)
      clients: 4064
      
      ```
      49876158
  17. 21 Jun, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix crash on RM_Call with script mode. (#10886) · 61baabd8
      Meir Shpilraien (Spielrein) authored
      The PR fixes 2 issues:
      
      ### RM_Call crash on script mode
      
      `RM_Call` can potentially be called from a background thread where `server.current_client`
      are not set. In such case we get a crash on `NULL` dereference.
      The fix is to check first if `server.current_client` is `NULL`, if it does we should
      verify disc errors and readonly replica as we do to any normal clients (no masters nor AOF).
      
      ### RM_Call block OOM commands when not needed
      
      Again `RM_Call` can be executed on a background thread using a `ThreadSafeCtx`.
      In such case `server.pre_command_oom_state` can be irrelevant and should not be
      considered when check OOM state. This cause OOM commands to be blocked when
      not necessarily needed.
      
      In such case, check the actual used memory (and not the cached value). Notice that in
      order to know if the cached value can be used, we check that the ctx that was used on
      the `RM_Call` is a ThreadSafeCtx. Module writer can potentially abuse the API and use
      ThreadSafeCtx on the main thread. We consider this as a API miss used.
      61baabd8
  18. 19 Jun, 2022 1 commit
  19. 14 Jun, 2022 3 commits
    • Oran Agra's avatar
      optimize zset conversion on large ZRANGESTORE (#10789) · 21891003
      Oran Agra authored
      when we know the size of the zset we're gonna store in advance,
      we can check if it's greater than the listpack encoding threshold,
      in which case we can create a skiplist from the get go, and avoid
      converting the listpack to skiplist later after it was already populated.
      21891003
    • Oran Agra's avatar
      Script that made modification will not break with unexpected NOREPLICAS error (#10855) · 8ef4f1db
      Oran Agra authored
      If a script made a modification and then was interrupted for taking too long.
      there's a chance redis will detect that a replica dropped and would like to reject
      write commands with NOREPLICAS due to insufficient good replicas.
      returning an error on a command in this case breaks the script atomicity.
      
      The same could in theory happen with READONLY, MISCONF, but i don't think
      these state changes can happen during script execution.
      8ef4f1db
    • Oran Agra's avatar
      Allow ECHO in loading and stale modes (#10853) · ffa00770
      Oran Agra authored
      I noticed that scripting.tcl uses INFO from within a script and thought it's an
      overkill and concluded it's nicer to use another CMD_STALE command,
      decided to use ECHO, and then noticed it's not at all allowed in stale mode.
      probably overlooked at #6843
      ffa00770
  20. 12 Jun, 2022 1 commit
    • Binbin's avatar
      Fixed SET and BITFIELD commands being wrongly marked movablekeys (#10837) · 92fb4f4f
      Binbin authored
      
      
      The SET and BITFIELD command were added `get_keys_function` in #10148, causing
      them to be wrongly marked movablekeys in `populateCommandMovableKeys`.
      
      This was an unintended side effect introduced in #10148 (7.0 RC1)
      which could cause some clients an extra round trip for these commands in cluster mode.
      
      Since we define movablekeys as a way to determine if the legacy range [first, last, step]
      doesn't find all keys, then we need a completely different approach.
      
      The right approach should be to check if the legacy range covers all key-specs,
      and if none of the key-specs have the INCOMPLETE flag. 
      This way, we don't need to look at getkeys_proc of VARIABLE_FLAG at all.
      Probably with the exception of modules, who may still not be using key-specs.
      
      In this PR, we removed `populateCommandMovableKeys` and put its logic in
      `populateCommandLegacyRangeSpec`.
      In order to properly serve both old and new modules, we must probably keep relying
      CMD_MODULE_GETKEYS, but do that only for modules that don't declare key-specs. 
      For ones that do, we need to take the same approach we take with native redis commands.
      
      This approach was proposed by Oran. Fixes #10833
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      92fb4f4f
  21. 09 Jun, 2022 1 commit
    • Christian Krieg's avatar
      Fixing test to consider statically linked binaries (#10835) · 032619b8
      Christian Krieg authored
      
      
      The test calls `ldd` on `redis-server` in order to find out whether the binary
      was linked against `libmusl`; However, `ldd` returns a value different from `0`
      when statically linking the binaries agains libc-musl, because `redis-server` is
      not a dynamic executable (as given by the exception thrown by the failing test),
      and `make test` terminates with an error::
      
         $ ldd src/redis-server
             not a dynamic executable
         $ echo $?
         1
      
      This commit fixes the test by ignoring such failures.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      032619b8
  22. 07 Jun, 2022 1 commit
    • Petr Vaněk's avatar
      Update musl libc detection pattern (#10826) · f22bfe86
      Petr Vaněk authored
      This change fixes failing `integration/logging.tcl` test in Gentoo with
      musl libc, where `ldd` returns
      ```
      libc.so => /lib/ld-musl-x86_64.so.1 (0x7f9d5f171000)
      ```
      unlike Alpine's
      ```
      libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f82cfa16000)
      ```
      The solution is to extend matching pattern introduced in #8532.
      f22bfe86
  23. 02 Jun, 2022 2 commits
    • zhaozhao.zz's avatar
      rewrite alias config to original name (#10811) · a18c91d6
      zhaozhao.zz authored
      
      
      Redis 7 adds some new alias config like `hash-max-listpack-entries` alias `hash-max-ziplist-entries`.
      
      If a config file contains both real name and alias like this:
      ```
      hash-max-listpack-entries 20
      hash-max-ziplist-entries 20
      ```
      
      after set `hash-max-listpack-entries` to 100 and `config rewrite`, the config file becomes to:
      ```
      hash-max-listpack-entries 100
      hash-max-ziplist-entries 20
      ```
      
      we can see that the alias config is not modified, and users will get wrong config after restart.
      
      6.0 and 6.2 doesn't have this bug, since they only have the `slave` word alias.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      a18c91d6
    • zhugezy's avatar
      Fix bugs in CONFIG REWRITE, omitting rename-command and include lines, and... · cf3323db
      zhugezy authored
      
      Fix bugs in CONFIG REWRITE, omitting rename-command and include lines, and inserting comments around module and acl configs (#10761)
      
      A regression from #10285 (redis 7.0).
      CONFIG REWRITE would put lines with: `include`, `rename-command`,
      `user`,  `loadmodule`, and any module specific config in a comment.
      
      For ACL `user`, `loadmodule` and module specific configs would be
      re-inserted at the end (instead of updating existing lines), so the only
      implication is a messy config file full of comments.
      
      But for `rename-command` and `include`, the implication would be that
      they're now missing, so a server restart would lose them.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      cf3323db
  24. 01 Jun, 2022 2 commits
    • Oran Agra's avatar
      Expose script flags to processCommand for better handling (#10744) · df558618
      Oran Agra authored
      The important part is that read-only scripts (not just EVAL_RO
      and FCALL_RO, but also ones with `no-writes` executed by normal EVAL or
      FCALL), will now be permitted to run during CLIENT PAUSE WRITE (unlike
      before where only the _RO commands would be processed).
      
      Other than that, some errors like OOM, READONLY, MASTERDOWN are now
      handled by processCommand, rather than the command itself affects the
      error string (and even error code in some cases), and command stats.
      
      Besides that, now the `may-replicate` commands, PFCOUNT and PUBLISH, will
      be considered `write` commands in scripts and will be blocked in all
      read-only scripts just like other write commands.
      They'll also be blocked in EVAL_RO (i.e. even for scripts without the
      `no-writes` shebang flag.
      
      This commit also hides the `may_replicate` flag from the COMMAND command
      output. this is a **breaking change**.
      
      background about may_replicate:
      We don't want to expose a no-may-replicate flag or alike to scripts, since we
      consider the may-replicate thing an internal concern of redis, that we may
      some day get rid of.
      In fact, the may-replicate flag was initially introduced to flag EVAL: since
      we didn't know what it's gonna do ahead of execution, before function-flags
      existed). PUBLISH and PFCOUNT, both of which because they have side effects
      which may some day be fixed differently.
      
      code changes:
      The changes in eval.c are mostly code re-ordering:
      - evalCalcFunctionName is extracted out of evalGenericCommand
      - evalExtractShebangFlags is extracted luaCreateFunction
      - evalGetCommandFlags is new code
      df558618
    • Oran Agra's avatar
      Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check... · b2061de2
      Oran Agra authored
      Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check bug, and new RM_Call checks. (#10786)
      
      * Fix broken protocol when redis can't persist to RDB (general commands, not
        modules), excessive newline. regression of #10372 (7.0 RC3)
      * Fix broken protocol when Redis can't persist to AOF (modules and
        scripts), missing newline.
      * Fix bug in OOM check of EVAL scripts called from RM_Call.
        set the cached OOM state for scripts before executing module commands too,
        so that it can serve scripts that are executed by modules.
        i.e. in the past EVAL executed by RM_Call could have either falsely
        fail or falsely succeeded because of a wrong cached OOM state flag.
      * Fix bugs with RM_Yield:
        1. SHUTDOWN should only accept the NOSAVE mode
        2. Avoid eviction during yield command processing.
        3. Avoid processing master client commands while yielding from another client
      * Add new two more checks to RM_Call script mode.
        1. READONLY You can't write against a read only replica
        2. MASTERDOWN Link with MASTER is down and `replica-serve-stale-data` is set to `no`
      * Add new RM_Call flag to let redis automatically refuse `deny-oom` commands
        while over the memory limit. 
      * Add tests to cover various errors from Scripts, Modules, Modules
        calling scripts, and Modules calling commands in script mode.
      
      Add tests:
      * Looks like the MISCONF error was completely uncovered by the tests,
        add tests for it, including from scripts, and modules
      * Add tests for NOREPLICAS from scripts
      * Add tests for the various errors in module RM_Call, including RM_Call that
        calls EVAL, and RM_call in "eval mode". that includes:
        NOREPLICAS, READONLY, MASTERDOWN, MISCONF
      b2061de2
  25. 31 May, 2022 1 commit
    • Harkrishn Patro's avatar
      Sharded pubsub publish messagebulk as smessage (#10792) · 4065b4f2
      Harkrishn Patro authored
      To easily distinguish between sharded channel message and a global
      channel message, introducing `smessage` (instead of `message`) as
      message bulk for sharded channel publish message.
      
      This is gonna be a breaking change in 7.0.1!
      
      Background:
      Sharded pubsub introduced in redis 7.0, but after the release we quickly
      realized that the fact that it's problematic that the client can't distinguish
      between normal (global) pubsub messages and sharded ones.
      This is important because the same connection can subscribe to both,
      but messages sent to one pubsub system are not propagated to the
      other (they're completely separate), so if one connection is used to
      subscribe to both, we need to assist the client library to know which
      message it got so it can forward it to the correct callback.
      4065b4f2
  26. 30 May, 2022 1 commit
  27. 29 May, 2022 1 commit
    • Binbin's avatar
      Fix sentinel disconnect test timing issue after auth-pass change (#10784) · 1013cbea
      Binbin authored
      There is a timing issue reported in test-sanitizer-address (gcc):
      ```
      Sentinels (re)connection following SENTINEL SET mymaster auth-pass:
      FAILED: Expected to be disconnected from master due to wrong password
      ```
      
      The reason we reach it, is because the test is fast enough to modify
      auth-pass and test sentinel connection status with the server,
      before its scheduled operation got the chance to update connection
      status with the server.
      
      We need to wait for `sentinelTimer` to kick in, and then update the
      connection status. Replace condition with wait_for_condition on the check.
      
      Fix just like #10480 did
      1013cbea
  28. 27 May, 2022 1 commit
    • Vitaly's avatar
      Fix ZRANGESTORE crash when zset_max_listpack_entries is 0 (#10767) · 6461f09f
      Vitaly authored
      When `zrangestore` is called container destination object is created. 
      Before this PR we used to create a listpack based object even if `zset-max-ziplist-entries`
      or equivalent`zset-max-listpack-entries` was set to 0.
      This triggered immediate conversion of the listpack into a skiplist in `zrangestore`, which hits
      an assertion resulting in an engine crash.
      
      Added a TCL test that reproduces this issue.
      6461f09f
  29. 26 May, 2022 1 commit