1. 12 Dec, 2022 5 commits
    • sundb's avatar
      Fix crash due to to reuse iterator entry after list deletion in module (#11383) · 9fc20f4f
      sundb authored
      
      
      In the module, we will reuse the list iterator entry for RM_ListDelete, but `listTypeDelete` will only update
      `quicklistEntry->zi` but not `quicklistEntry->node`, which will result in `quicklistEntry->node` pointing to
      a freed memory address if the quicklist node is deleted.
      
      This PR sync `key->u.list.index` and `key->u.list.entry` to list iterator after `RM_ListDelete`.
      
      This PR also optimizes the release code of the original list iterator.
      Co-authored-by: default avatarViktor Söderqvist <viktor@zuiderkwast.se>
      (cherry picked from commit 6dd21355)
      9fc20f4f
    • C Charles's avatar
      MIGTATE with AUTH that contains "keys" is getting wrong key names in... · f95af778
      C Charles authored
      MIGTATE with AUTH that contains "keys" is getting wrong key names in migrateGetKeys, leads to ACL errors (#11253)
      
      When using the MIGRATE, with a destination Redis that has the user name or password set to the string "keys",
      Redis would have determine the wrong set of key names the command is gonna access.
      This lead to ACL returning wrong authentication result.
      
      Destination instance:
      ```
      127.0.0.1:6380> acl setuser default >keys
      OK
      127.0.0.1:6380> acl setuser keys on nopass ~* &* +@all
      OK
      ```
      
      Source instance:
      ```
      127.0.0.1:6379> set a 123
      OK
      127.0.0.1:6379> acl setuser cc on nopass ~a* +@all
      OK
      127.0.0.1:6379> auth cc 1
      OK
      127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth keys keys a
      (error) NOPERM this user has no permissions to access one of the keys used as arguments
      127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth2 keys pswd keys a
      (error) NOPERM this user has no permissions to access one of the keys used as arguments
      ```
      
      Using `acl dryrun` we know that the parameters of `auth` and `auth2` are mistaken for the `keys` option.
      ```
      127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth keys keys a
      "This user has no permissions to access the 'keys' key"
      127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth2 keys pswd keys a
      "This user has no permissions to access the 'pswd' key"
      ```
      
      Fix the bug by editing db.c/migrateGetKeys function, which finds the `keys` option and all the keys following.
      
      (cherry picked from commit 9ab873d9)
      f95af778
    • Meir Shpilraien (Spielrein)'s avatar
      `RedisModule_ResetDataset` should not clear the functions. (#11268) · 6cf24fa4
      Meir Shpilraien (Spielrein) authored
      As mentioned on docs, `RM_ResetDataset` Performs similar operation to FLUSHALL.
      As FLUSHALL do not clean the function, `RM_ResetDataset` should not clean the functions
      as well.
      
      (cherry picked from commit d2ad01ab)
      6cf24fa4
    • guybe7's avatar
      RM_CreateCommand should not set CMD_KEY_VARIABLE_FLAGS automatically (#11320) · 5b2119c6
      guybe7 authored
      The original idea behind auto-setting the default (first,last,step) spec was to use
      the most "open" flags when the user didn't provide any key-spec flags information.
      
      While the above idea is a good approach, it really makes no sense to set
      CMD_KEY_VARIABLE_FLAGS if the user didn't provide the getkeys-api flag:
      in this case there's not way to retrieve these variable flags, so what's the point?
      
      Internally in redis there was code to ignore this already, so this fix doesn't change
      redis's behavior, it only affects the output of COMMAND command.
      
      (cherry picked from commit 3330ea18)
      5b2119c6
    • Shaya Potter's avatar
      Add RM_SetContextUser to support acl validation in RM_Call (and scripts) (#10966) · b8fcd322
      Shaya Potter authored
      Adds a number of user management/ACL validaiton/command execution functions to improve a
      Redis module's ability to enforce ACLs correctly and easily.
      
      * RM_SetContextUser - sets a RedisModuleUser on the context, which RM_Call will use to both
        validate ACLs (if requested and set) as well as assign to the client so that scripts executed via
        RM_Call will have proper ACL validation.
      * RM_SetModuleUserACLString - Enables one to pass an entire ACL string, not just a single OP
        and have it applied to the user
      * RM_GetModuleUserACLString - returns a stringified version of the user's ACL (same format as dump
        and list).  Contains an optimization to cache the stringified version until the underlying ACL is modified.
      * Slightly re-purpose the "C" flag to RM_Call from just being about ACL check before calling the
        command, to actually running the command with the right user, so that it also affects commands
        inside EVAL scripts. see #11231
      
      (cherry picked from commit 6e993a5d)
      b8fcd322
  2. 21 Sep, 2022 14 commits
    • Oran Agra's avatar
      Fix heap overflow vulnerability in XAUTOCLAIM (CVE-2022-35951) · fa6815e1
      Oran Agra authored
      Executing an XAUTOCLAIM command on a stream key in a specific state, with a
      specially crafted COUNT argument may cause an integer overflow, a subsequent
      heap overflow, and potentially lead to remote code execution.
      The problem affects Redis versions 7.0.0 or newer.
      fa6815e1
    • Shaya Potter's avatar
      Improve cmd_flags for script/functions in RM_Call (#11159) · a1ec0cae
      Shaya Potter authored
      When RM_Call was used with `M` (reject OOM), `W` (reject writes),
      as well as `S` (rejecting stale or write commands in "Script mode"),
      it would have only checked the command flags, but not the declared
      script flag in case it's a command that runs a script.
      
      Refactoring: extracts out similar code in server.c's processCommand
      to be usable in RM_Call as well.
      
      (cherry picked from commit bed6d759)
      a1ec0cae
    • Madelyn Olson's avatar
      Cluster test infra (taken from #10920) · b8beda61
      Madelyn Olson authored
      * Taking just the test infrastrucutre from that commit.
      
      (cherry picked from commit 8a4e3bcd)
      b8beda61
    • Shay Fadida's avatar
      Fix missing sections for INFO ALL with module (#11291) · c9eabbf9
      Shay Fadida authored
      
      
      When using `INFO ALL <section>`, when `section` is a specific module section. 
      Redis will not print the additional section(s).
      
      The fix in this case, will search the modules info sections if the user provided additional sections to `ALL`.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit eedb8b17)
      c9eabbf9
    • sundb's avatar
      Fix crash due to delete entry from compress quicklistNode and wrongly split quicklistNode (#11242) · 01358df3
      sundb authored
      This PR mainly deals with 2 crashes introduced in #9357,
      and fix the QUICKLIST-PACKED-THRESHOLD mess in external test mode.
      
      1. Fix crash due to deleting an entry from a compress quicklistNode
         When inserting a large element, we need to create a new quicklistNode first,
         and then delete its previous element, if the node where the deleted element is
         located is compressed, it will cause a crash.
         Now add `dont_compress` to quicklistNode, if we want to use a quicklistNode
         after some operation, we can use this flag like following:
      
          ```c
          node->dont_compress = 1; /* Prevent to be compressed */
          some_operation(node); /* This operation might try to compress this node */
          some_other_operation(node); /* We can use this node without decompress it */
          node->dont_compress = 0; /* Re-able compression */
          quicklistCompressNode(node);
          ```
      
         Perhaps in the future, we could just disable the current entry from being
         compressed during the iterator loop, but that would require more work.
      
      2. Fix crash due to wrongly split quicklist
         before #9357, the offset param of _quicklistSplitNode() will not negative.
         For now, when offset is negative, the split extent will be wrong.
         following example:
          ```c
          int orig_start = after ? offset + 1 : 0;
          int orig_extent = after ? -1 : offset;
          int new_start = after ? 0 : offset;
          int new_extent = after ? offset + 1 : -1;
          # offset: -2, after: 1, node->count: 2
          # current wrong range: [-1,-1] [0,-1]
          # correct range: [1,-1] [0, 1]
          ```
      
         Because only `_quicklistInsert()` splits the quicklistNode and only
         `quicklistInsertAfter()`, `quicklistInsertBefore()` call _quicklistInsert(), 
         so `quicklistReplaceEntry()` and `listTypeInsert()` might occur this crash.
         But the iterator of `listTypeInsert()` is alway from head to tail(iter->offset is
         always positive), so it is not affected.
         The final conclusion is this crash only occur when we insert a large element
         with negative index into a list, that affects `LSET` command and `RM_ListSet`
         module api.
           
      3. In external test mode, we need to restore quicklist packed threshold after
         when the end of test.
      4. Show `node->count` in quicklistRepr().
      5. Add new tcl proc `config_get_set` to support restoring config in tests.
      
      (cherry picked from commit 13d25dd9)
      01358df3
    • chendianqiang's avatar
      Correctly handle scripts with shebang (not read-only) on a cluster replica (#11223) · 8c702f8d
      chendianqiang authored
      
      
      EVAL scripts are by default not considered `write` commands, so they were allowed on a replica.
      But when adding a shebang, they become `write` command (unless the `no-writes` flag is added).
      With this change we'll handle them as write commands, and reply with MOVED instead of
      READONLY when executed on a redis cluster replica.
      Co-authored-by: default avatarchendianqiang <chendianqiang@meituan.com>
      (cherry picked from commit e42d98ed)
      8c702f8d
    • Oran Agra's avatar
      Fix assertion when a key is lazy expired during cluster key migration (#11176) · a2a28b80
      Oran Agra authored
      Redis 7.0 has #9890 which added an assertion when the propagation queue
      was not flushed and we got to beforeSleep.
      But it turns out that when processCommands calls getNodeByQuery and
      decides to reject the command, it can lead to a key that was lazy
      expired and is deleted without later flushing the propagation queue.
      
      This change prevents lazy expiry from deleting the key at this stage
      (not as part of a command being processed in `call`)
      
      (cherry picked from commit c789fb0a)
      a2a28b80
    • Binbin's avatar
      Fix memory leak in moduleFreeCommand (#11147) · fa60cd13
      Binbin authored
      Currently, we call zfree(cmd->args), but the argument array
      needs to be freed recursively (there might be sub-args).
      Also fixed memory leaks on cmd->tips and cmd->history.
      
      Fixes #11145
      
      (cherry picked from commit fc3956e8)
      fa60cd13
    • DarrenJiang13's avatar
      fix the client type in trackingInvalidateKey() (#11052) · 4dfc487f
      DarrenJiang13 authored
      Fix bug with scripts ignoring client tracking NOLOOP and
      send an invalidation message anyway.
      
      (cherry picked from commit 44859a41)
      4dfc487f
    • Huang Zhw's avatar
      acl: bitfield with get and set|incrby can be executed with readonly permission (#11086) · ab524282
      Huang Zhw authored
      `bitfield` with `get` may not be readonly.
      
      ```
      127.0.0.1:6384> acl setuser hello on nopass %R~* +@all
      
      
      OK
      127.0.0.1:6384> auth hello 1
      OK
      127.0.0.1:6384> bitfield hello set i8 0 1
      (error) NOPERM this user has no permissions to access one of the keys used as arguments
      127.0.0.1:6384> bitfield hello set i8 0 1 get i8 0
      1) (integer) 0
      2) (integer) 1
      ```
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit ec5034a2)
      ab524282
    • Wen Hui's avatar
      Fix function load error message (#10964) · d1b3276b
      Wen Hui authored
      Update error messages for function load
      
      (cherry picked from commit beb9746a)
      d1b3276b
    • Huang Zhw's avatar
      tracking pending invalidation message of flushdb sent by (#11068) · 5632bbdc
      Huang Zhw authored
      trackingHandlePendingKeyInvalidations should use proto.
      
      (cherry picked from commit 61451b02)
      5632bbdc
    • Huang Zhw's avatar
      When client tracking is on, invalidation message of flushdb in a (#11038) · ba6ed445
      Huang Zhw authored
      When FLUSHDB / FLUSHALL / SWAPDB is inside MULTI / EXEC, the
      client side tracking invalidation message was interleaved with transaction response.
      
      
      (cherry picked from commit 6f0a27e3)
      ba6ed445
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #11030, use lua_rawget to avoid triggering metatables and crash. (#11032) · ddf1bcf7
      Meir Shpilraien (Spielrein) authored
      Fix #11030, use lua_rawget to avoid triggering metatables.
      
      #11030 shows how return `_G` from the Lua script (either function or eval), cause the
      Lua interpreter to Panic and the Redis processes to exit with error code 1.
      Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older
      versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable
      such that the metatable raises an error.
      
      The following example demonstrate the issue:
      ```
      127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0
      Error: Server closed the connection
      ```
      ```
      PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo')
      ```
      
      The Lua panic happened because when returning the result to the client, Redis needs to
      introspect the returning table and transform the table into a resp. In order to scan the table,
      Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error.
      This code is not running inside `pcall` (Lua protected call), so raising an error causes the
      Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1.
      
      Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable
      that raises error when trying to fetch a none existing key.
      
      ### Solution
      
      Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget`
      that simply return the value from the table without triggering any metatable logic.
      This is promised not to raise and error.
      
      The downside of this solution is that it might be considered as breaking change, if someone
      rely on metatable in the returned value. An alternative solution is to wrap this entire logic
      with `pcall` (Lua protected call), this alternative require a much bigger refactoring.
      
      ### Back Porting
      
      The same fix will work on older versions as well (6.2, 6.0). Notice that on those version,
      the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses
      Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done
      from inside the script itself.
      
      ### Tests
      
      Tests was added the verify the fix
      
      (cherry picked from commit 020e046b)
      ddf1bcf7
  3. 18 Jul, 2022 1 commit
    • Oran Agra's avatar
      Fix heap overflow corruption in XAUTOCLAIM (CVE-2022-31144) (#11002) · 15ae4e29
      Oran Agra authored
      The temporary array for deleted entries reply of XAUTOCLAIM was
      insufficient, but also in fact the COUNT argument should be used to
      control the size of the reply, so instead of terminating the loop by
      only counting the claimed entries, we'll count deleted entries as well.
      
      Fix #10968
      Addresses CVE-2022-31144
      
      (cherry picked from commit 2825b605)
      15ae4e29
  4. 11 Jul, 2022 3 commits
    • Binbin's avatar
      Trying to fix cluster test (#10963) · 693acc01
      Binbin authored
      #10942 break the new test added in #10449
      ```
      Testing unit: 29-slot-migration-response.tcl
      Cluster Join and auto-discovery test: FAILED: Cluster failed to join into a full mesh.
      ```
      
      It looks like we need to wait for the cluster in 28 to become stable.
      693acc01
    • Binbin's avatar
      Add cluster-port support to redis-cli --cluster (#10344) · 35e8ae3e
      Binbin authored
      
      
      In #9389, we add a new `cluster-port` config and make cluster bus port configurable,
      and currently redis-cli --cluster create/add-node doesn't support with a configurable `cluster-port` instance.
      Because redis-cli uses the old way (port + 10000) to send the `CLUSTER MEET` command.
      
      Now we add this support on redis-cli `--cluster`, note we don't need to explicitly pass in the
      `cluster-port` parameter, we can get the real `cluster-port` of the node in `clusterManagerNodeLoadInfo`,
      so the `--cluster create` and `--cluster add-node` interfaces have not changed.
      
      We will use the `cluster-port` when we are doing `CLUSTER MEET`, also note that `CLUSTER MEET` bus-port
      parameter was added in 4.0, so if the bus_port (the one in redis-cli) is 0, or equal (port + 10000),
      we just call `CLUSTER MEET` with 2 arguments, using the old form.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      35e8ae3e
    • Madelyn Olson's avatar
      Fix crash during handshake and cluster shards call (#10942) · e6a1b2ea
      Madelyn Olson authored
      * Fix an engine crash when there are nodes in handshaking and a user calls cluster shards
      e6a1b2ea
  5. 04 Jul, 2022 2 commits
  6. 03 Jul, 2022 1 commit
  7. 30 Jun, 2022 1 commit
    • Binbin's avatar
      Add SENTINEL command flag to CLIENT/COMMANDS subcommands (#10904) · 35e836c2
      Binbin authored
      This was harmless because we marked the parent command
      with SENTINEL flag. So the populateCommandTable was ok.
      And we also don't show the flag (SENTINEL and ONLY-SENTNEL)
      in COMMAND INFO.
      
      In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL
      flags check when populating the sub-commands.
      so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
      35e836c2
  8. 29 Jun, 2022 1 commit
  9. 28 Jun, 2022 1 commit
    • jonnyomerredis's avatar
      Add sharded pubsub keychannel count to client info (#10895) · 35c2ee87
      jonnyomerredis authored
      When calling CLIENT INFO/LIST, and in various debug prints, Redis is printing
      the number of pubsub channels / patterns the client is subscribed to.
      With the addition of sharded pubsub, it would be useful to print the number of
      keychannels the client is subscribed to as well.
      35c2ee87
  10. 27 Jun, 2022 1 commit
    • Viktor Söderqvist's avatar
      Add missing REDISMODULE_CLIENTINFO_INITIALIZER (#10885) · 6af02100
      Viktor Söderqvist authored
      The module API docs mentions this macro, but it was not defined (so no one could have used it).
      
      Instead of adding it as is, we decided to add a _V1 macro, so that if / when we some day extend this struct,
      modules that use this API and don't need the extra fields, will still use the old version
      and still be compatible with older redis version (despite being compiled with newer redismodule.h)
      6af02100
  11. 26 Jun, 2022 3 commits
    • RinChanNOW!'s avatar
      Support conversion between `RedisModuleString` and `unsigned long long` (#10889) · 28546373
      RinChanNOW! authored
      
      
      Since the ranges of `unsigned long long` and `long long` are different, we cannot read an
      `unsigned long long` integer from a `RedisModuleString` by `RedisModule_StringToLongLong` . 
      
      So I added two new Redis Module APIs to support the conversion between these two types:
      * `RedisModule_StringToULongLong`
      * `RedisModule_CreateStringFromULongLong`
      Signed-off-by: default avatarRinChanNOWWW <hzy427@gmail.com>
      28546373
    • Binbin's avatar
      redis-server command line arguments allow passing config name and value in the same arg (#10866) · d443e312
      Binbin authored
      This commit has two topics.
      
      ## Passing config name and value in the same arg
      In #10660 (Redis 7.0.1), when we supported the config values that can start with `--` prefix (one of the two topics of that PR),
      we broke another pattern: `redis-server redis.config "name value"`, passing both config name
      and it's value in the same arg, see #10865
      
      This wasn't a intended change (i.e we didn't realize this pattern used to work).
      Although this is a wrong usage, we still like to fix it.
      
      Now we support something like:
      ```
      src/redis-server redis.conf "--maxmemory '700mb'" "--maxmemory-policy volatile-lru" --proc-title-template --my--title--template --loglevel verbose
      ```
      
      ## Changes around --save
      Also in this PR, we undo the breaking change we made in #10660 on purpose.
      1. `redis-server redis.conf --save --loglevel verbose` (missing `save` argument before anotehr argument).
          In 7.0.1, it was throwing...
      d443e312
    • Viktor Söderqvist's avatar
      Add RM_SetClientNameById and RM_GetClientNameById (#10839) · 6272ca60
      Viktor Söderqvist authored
      Adding Module APIs to let the module read and set the client name of an arbitrary connection.
      6272ca60
  12. 23 Jun, 2022 1 commit
    • judeng's avatar
      fix benchmark failure in daily test with TLS (#10896) · d2405b9b
      judeng authored
      The new test added in #10891 can fail with a different error.
      see comment in networking.c saying
      ```c
              /* That's a best effort error message, don't check write errors.
               * Note that for TLS connections, no handshake was done yet so nothing
               * is written and the connection will just drop. */
      ```
      d2405b9b
  13. 22 Jun, 2022 1 commit
    • judeng's avatar
      fix redis-benchmark's bug: check if clients are created successfully in idle mode (#10891) · 49876158
      judeng authored
      my maxclients config:
      ```
      redis-cli config get maxclients
      1) "maxclients"
      2) "4064"
      ```
      
      Before this bug was fixed, creating 4065 clients appeared to be successful, but only 4064 were actually created```
      ```
      ./redis-benchmark -c 4065 -I
      Creating 4065 idle connections and waiting forever (Ctrl+C when done)
      cients: 4065
      ```
      
      now :
      ```
      ./redis-benchmark -c 4065 -I
      Creating 4065 idle connections and waiting forever (Ctrl+C when done)
      Error from server: ERR max number of clients reached
      
      ./redis-benchmark -c 4064 -I
      Creating 4064 idle connections and waiting forever (Ctrl+C when done)
      clients: 4064
      
      ```
      49876158
  14. 21 Jun, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix crash on RM_Call with script mode. (#10886) · 61baabd8
      Meir Shpilraien (Spielrein) authored
      The PR fixes 2 issues:
      
      ### RM_Call crash on script mode
      
      `RM_Call` can potentially be called from a background thread where `server.current_client`
      are not set. In such case we get a crash on `NULL` dereference.
      The fix is to check first if `server.current_client` is `NULL`, if it does we should
      verify disc errors and readonly replica as we do to any normal clients (no masters nor AOF).
      
      ### RM_Call block OOM commands when not needed
      
      Again `RM_Call` can be executed on a background thread using a `ThreadSafeCtx`.
      In such case `server.pre_command_oom_state` can be irrelevant and should not be
      considered when check OOM state. This cause OOM commands to be blocked when
      not necessarily needed.
      
      In such case, check the actual used memory (and not the cached value). Notice that in
      order to know if the cached value can be used, we check that the ctx that was used on
      the `RM_Call` is a ThreadSafeCtx. Module writer can potentially abuse the API and use
      ThreadSafeCtx on the main thread. We consider this as a API miss used.
      61baabd8
  15. 19 Jun, 2022 1 commit
  16. 14 Jun, 2022 3 commits
    • Oran Agra's avatar
      optimize zset conversion on large ZRANGESTORE (#10789) · 21891003
      Oran Agra authored
      when we know the size of the zset we're gonna store in advance,
      we can check if it's greater than the listpack encoding threshold,
      in which case we can create a skiplist from the get go, and avoid
      converting the listpack to skiplist later after it was already populated.
      21891003
    • Oran Agra's avatar
      Script that made modification will not break with unexpected NOREPLICAS error (#10855) · 8ef4f1db
      Oran Agra authored
      If a script made a modification and then was interrupted for taking too long.
      there's a chance redis will detect that a replica dropped and would like to reject
      write commands with NOREPLICAS due to insufficient good replicas.
      returning an error on a command in this case breaks the script atomicity.
      
      The same could in theory happen with READONLY, MISCONF, but i don't think
      these state changes can happen during script execution.
      8ef4f1db
    • Oran Agra's avatar
      Allow ECHO in loading and stale modes (#10853) · ffa00770
      Oran Agra authored
      I noticed that scripting.tcl uses INFO from within a script and thought it's an
      overkill and concluded it's nicer to use another CMD_STALE command,
      decided to use ECHO, and then noticed it's not at all allowed in stale mode.
      probably overlooked at #6843
      ffa00770