1. 08 Feb, 2022 4 commits
    • Wen Hui's avatar
      Make INFO command variadic (#6891) · 2e1bc942
      Wen Hui authored
      
      
      This is an enhancement for INFO command, previously INFO only support one argument
      for different info section , if user want to get more categories information, either perform
      INFO all / default or calling INFO for multiple times.
      
      **Description of the feature**
      
      The goal of adding this feature is to let the user retrieve multiple categories via the INFO
      command, and still avoid emitting the same section twice.
      
      A use case for this is like Redis Sentinel, which periodically calling INFO command to refresh
      info from monitored Master/Slaves, only Server and Replication part categories are used for
      parsing information. If the INFO command can return just enough categories that client side
      needs, it can save a lot of time for client side parsing it as well as network bandwidth.
      
      **Implementation**
      To share code between redis, sentinel, and other users of INFO (DEBUG and modules),
      we have a new `genInfoSectionDict` function that returns a dict and some boolean flags
      (e.g. `all`) to the caller (built from user input).
      Sentinel is later purging unwanted sections from that, and then it is forwarded to the info `genRedisInfoString`.
      
      **Usage Examples**
      INFO Server Replication   
      INFO CPU Memory
      INFO default commandstats
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2e1bc942
    • yoav-steinberg's avatar
      Consistent erros returned from EVAL scripts (#10218) · b76016a9
      yoav-steinberg authored
      This PR handles inconsistencies in errors returned from lua scripts.
      Details of the problem can be found in #10165.
      
      ### Changes
      
      - Remove double stack trace. It's enough that a stack trace is automatically added by the engine's error handler
        see https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/function_lua.c#L472-L485
        and https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/eval.c#L243-L255
      - Make sure all errors a preceded with an error code. Passing a simple string to `luaPushError()` will prepend it
        with a generic `ERR` error code.
      - Make sure lua error table doesn't include a RESP `-` error status. Lua stores redis error's as a lua table with a
        single `err` field and a string. When the string is translated back to RESP we add a `-` to it.
        See https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/script_lua.c#L510-L517
        So there's no need to store it in the lua table.
      
      ### Before & After
      ```diff
      --- <unnamed>
      +++ <unnamed>
      @@ -1,14 +1,14 @@
        1: config set maxmemory 1
        2: +OK
        3: eval "return redis.call('set','x','y')" 0
      - 4: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: @user_script: 1: -OOM command not allowed when used memory > 'maxmemory'.
      + 4: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: OOM command not allowed when used memory > 'maxmemory'.
        5: eval "return redis.pcall('set','x','y')" 0
      - 6: -@user_script: 1: -OOM command not allowed when used memory > 'maxmemory'.
      + 6: -OOM command not allowed when used memory > 'maxmemory'.
        7: eval "return redis.call('select',99)" 0
        8: -ERR Error running script (call to 4ad5abfc50bbccb484223905f9a16f09cd043ba8): @user_script:1: ERR DB index is out of range
        9: eval "return redis.pcall('select',99)" 0
       10: -ERR DB index is out of range
       11: eval_ro "return redis.call('set','x','y')" 0
      -12: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: @user_script: 1: Write commands are not allowed from read-only scripts.
      +12: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: ERR Write commands are not allowed from read-only scripts.
       13: eval_ro "return redis.pcall('set','x','y')" 0
      -14: -@user_script: 1: Write commands are not allowed from read-only scripts.
      +14: -ERR Write commands are not allowed from read-only scripts.
      ```
      b76016a9
    • guybe7's avatar
      X[AUTO]CLAIM should skip deleted entries (#10227) · 3c3e6cc1
      guybe7 authored
      Fix #7021 #8924 #10198
      
      # Intro
      Before this commit X[AUTO]CLAIM used to transfer deleted entries from one
      PEL to another, but reply with "nil" for every such entry (instead of the entry id).
      The idea (for XCLAIM) was that the caller could see this "nil", realize the entry
      no longer exists, and XACK it in order to remove it from PEL.
      The main problem with that approach is that it assumes there's a correlation
      between the index of the "id" arguments and the array indices, which there
      isn't (in case some of the input IDs to XCLAIM never existed/read):
      
      ```
      127.0.0.1:6379> XADD x 1 f1 v1
      "1-0"
      127.0.0.1:6379> XADD x 2 f1 v1
      "2-0"
      127.0.0.1:6379> XADD x 3 f1 v1
      "3-0"
      127.0.0.1:6379> XGROUP CREATE x grp 0
      OK
      127.0.0.1:6379> XREADGROUP GROUP grp Alice COUNT 2 STREAMS x >
      1) 1) "x"
         2) 1) 1) "1-0"
               2) 1) "f1"
                  2) "v1"
            2) 1) "2-0"
               2) 1) "f1"
                  2) "v1"
      127.0.0.1:6379> XDEL x 1 2
      (integer) 2
      127.0.0.1:6379> XCLAIM x grp Bob 0 0-99 1-0 1-99 2-0
      1) (nil)
      2) (nil)
      ```
      
      # Changes
      Now,  X[AUTO]CLAIM acts in the following way:
      1. If one tries to claim a deleted entry, we delete it from the PEL we found it in
        (and the group PEL too). So de facto, such entry is not claimed, just cleared
        from PEL (since anyway it doesn't exist in the stream)
      2. since we never claim deleted entries, X[AUTO]CLAIM will never return "nil"
        instead of an entry.
      3. add a new element to XAUTOCLAIM's response (see below)
      
      # Knowing which entries were cleared from the PEL
      The caller may want to log any entries that were found in a PEL but deleted from
      the stream itself (it would suggest that there might be a bug in the application:
      trimming the stream while some entries were still no processed by the consumers)
      
      ## XCLAIM
      the set {XCLAIM input ids} - {XCLAIM returned ids} contains all the entry ids that were
      not claimed which means they were deleted (assuming the input contains only entries
      from some PEL). The user doesn't need to XACK them because XCLAIM had already
      deleted them from the source PEL.
      
      ## XAUTOCLAIM
      XAUTOCLAIM has a new element added to its reply: it's an array of all the deleted
      stream IDs it stumbled upon.
      
      This is somewhat of a breaking change since X[AUTO]CLAIM used to be able to reply
      with "nil" and now it can't... But since it was undocumented (and generally a bad idea
      to rely on it, as explained above) the breakage is not that bad.
      3c3e6cc1
    • Oran Agra's avatar
      Handle key-spec flags with modules (#10237) · 66be30f7
      Oran Agra authored
      - add COMMAND GETKEYSANDFLAGS sub-command
      - add RM_KeyAtPosWithFlags and GetCommandKeysWithFlags
      - RM_KeyAtPos and RM_CreateCommand set flags requiring full access for keys
      - RM_CreateCommand set VARIABLE_FLAGS
      - expose `variable_flags` flag in COMMAND INFO key-specs
      - getKeysFromCommandWithSpecs prefers key-specs over getkeys-api
      - add tests for all of these
      66be30f7
  2. 07 Feb, 2022 7 commits
    • ivanstosic-janea's avatar
      Fix protocol error caused by redis-benchmark (#10236) · bb875603
      ivanstosic-janea authored
      The protocol error was caused by the buggy `writeHandler` in `redis-benchmark.c`,
      which didn't handle one of the cases, thereby repeating data, leading to protocol errors
      when the values being sent are very long.
      
      This PR fixes #10233, issue introduced by #7959
      bb875603
    • Avital-Fine's avatar
    • Binbin's avatar
      COMMAND DOCS avoid adding summary/since if they don't exist (#10252) · 7f4cca11
      Binbin authored
      If summary or since is empty, we used to return NULL in
      COMMAND DOCS. Currently all redis commands will have these
      two fields.
      
      But not for module command, summary and since are optional
      for RM_SetCommandInfo. With the change in #10043, if a module
      command doesn't have the summary or since, redis-cli will
      crash (see #10250).
      
      In this commit, COMMAND DOCS avoid adding summary or since
      when they are missing.
      7f4cca11
    • Binbin's avatar
      Fix redis-cli with sentinel crash due to SENTINEL DEBUG missing summary (#10250) · b95beeb5
      Binbin authored
      Fix redis-cli with sentinel crash due to SENTINEL DEBUG missing summary
      
      Because SENTINEL DEBUG missing summary in its json file,
      with the change in #10043, the following assertion will fail.
      ```
      [redis]# src/redis-cli -p 26379
      redis-cli: redis-cli.c:678: cliInitCommandHelpEntry: Assertion `reply->type == 1' failed.
      ```
      
      This commit add the summary and complexity for SENTINEL DEBUG,
      which introduced in #9291, and also improved the help message.
      b95beeb5
    • yoav-steinberg's avatar
      acl check api for functions and eval (#10220) · 9dfeda58
      yoav-steinberg authored
      Changes:
      1. Adds the `redis.acl_check_cmd()` api to lua scripts. It can be used to check if the
        current user has permissions to execute a given command. The new function receives
        the command to check as an argument exactly like `redis.call()` receives the command
        to execute as an argument.
      2. In the PR I unified the code used to convert lua arguments to redis argv arguments from
        both the new `redis.acl_check_cmd()` API and the `redis.[p]call()` API. This cleans up
        potential duplicate code.
      3. While doing the refactoring in 2 I noticed there's an optimization to reduce allocation calls
        when parsing lua arguments into an `argv` array in the `redis.[p]call()` implementation.
        These optimizations were introduced years ago in 48c49c48
        and 4f686555. It is unclear why this was added.
        The original commit message claims a 4% performance increase which I couldn't recreate
        and might not be worth it even if it did recreate. This PR removes that optimization.
        Following are details of the benchmark I did that couldn't reveal any performance
        improvements due to this optimization:
      
      ```
      benchmark 1: src/redis-benchmark -P 500 -n 10000000 eval 'return redis.call("ping")' 0
      benchmark 2: src/redis-benchmark -P 500 -r 1000 -n 1000000 eval 'return redis.call("mset","k1__rand_int__","v1__rand_int__","k2__rand_int__","v2__rand_int__","k3__rand_int__","v3__rand_int__","k4__rand_int__","v4__rand_int__")' 0
      benchmark 3: src/redis-benchmark -P 500 -r 1000 -n 100000 eval "for i=1,100,1 do redis.call('set','kk'..i,'vv'..__rand_int__) end return redis.call('get','kk5')" 0
      benchmark 4: src/redis-benchmark -P 500 -r 1000 -n 1000000 eval 'return redis.call("mset","k1__rand_int__","v1__rand_int__","k2__rand_int__","v2__rand_int__","k3__rand_int__","v3__rand_int__","k4__rand_int__","v4__rand_int__xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")'
      ```
      I ran the benchmark on this branch with and without commit 68b71680a4d3bb8f0509e06578a9f15d05b92a47
      Results in requests per second:
      cmd | without optimization | without optimization 2nd run | with original optimization | with original optimization 2nd run
      -- | -- | -- | -- | --
      1 | 461233.34 | 477395.31 | 471098.16 | 469946.91
      2 | 34774.14 | 35469.8 | 35149.38 | 34464.93
      3 | 6390.59 | 6281.41 | 6146.28 | 6464.12
      4 | 28005.71 |   | 27965.77 |  
      
      As you can see, different use cases showed identical or negligible performance differences.
      So finally I decided to chuck the original optimization and simplify the code.
      9dfeda58
    • Binbin's avatar
      create-cluster clean now will clean appendonlydir (#10223) · c5e3d135
      Binbin authored
      In #9788, now we stores all persistent append-only files in
      a dedicated directory. The name of the directory is determined
      by the appenddirname configuration parameter in redis.conf.
      Now each node have a separate folder.
      Update create-cluster clean to clean this default directory.
      c5e3d135
    • weiguo's avatar
      d6e9cde5
  3. 06 Feb, 2022 4 commits
  4. 05 Feb, 2022 3 commits
  5. 04 Feb, 2022 3 commits
    • Viktor Söderqvist's avatar
      Command info module API (#10108) · 0a82fe84
      Viktor Söderqvist authored
      Adds RM_SetCommandInfo, allowing modules to provide the following command info:
      
      * summary
      * complexity
      * since
      * history
      * hints
      * arity
      * key specs
      * args
      
      This information affects the output of `COMMAND`, `COMMAND INFO` and `COMMAND DOCS`,
      Cluster, ACL and is used to filter commands with the wrong number of arguments before
      the call reaches the module code.
      
      The recently added API functions for key specs (never released) are removed.
      
      A minimalist example would look like so:
      ```c
          RedisModuleCommand *mycmd = RedisModule_GetCommand(ctx,"mymodule.mycommand");
          RedisModuleCommandInfo mycmd_info = {
              .version = REDISMODULE_COMMAND_INFO_VERSION,
              .arity = -5,
              .summary = "some description",
          };
          if (RedisModule_SetCommandInfo(mycmd, &mycmd_info) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      ````
      
      Notes:
      * All the provided information (including strings) is copied, not keeping references to the API input data.
      * The version field is actually a static struct that contains the sizes of the the structs used in arrays,
        so we can extend these in the future and old version will still be able to take the part they can support.
      0a82fe84
    • Binbin's avatar
      Fix SENTINEL SET config rewrite test (#10232) · d7fcb3c5
      Binbin authored
      Change the sentinel config file to a directory in SENTINEL SET test.
      So it will now fail on the `rename` in `rewriteConfigOverwriteFile`.
      
      The test used to set the sentinel config file permissions to `000` to
      simulate failure. But it fails on centos7 / freebsd / alpine. (introduced in #10151)
      
      Other changes:
      1. More error messages after the config rewrite failure.
      2. Modify arg name `force_all` in `rewriteConfig` to `force_write`. (was rename in #9304)
      3. Fix a typo in debug quicklist-packed-threshold, then -> than. (#9357)
      d7fcb3c5
    • Binbin's avatar
      Fix cluster tests failing due to subcommand names (#10231) · d2fde2f6
      Binbin authored
      Introduced in #10128
      d2fde2f6
  6. 03 Feb, 2022 4 commits
  7. 02 Feb, 2022 2 commits
  8. 01 Feb, 2022 2 commits
  9. 31 Jan, 2022 3 commits
  10. 30 Jan, 2022 8 commits
    • Oran Agra's avatar
      Revent the attempt to fix cluster rebalance test (#10207) (#10212) · d364ede5
      Oran Agra authored
      It seems that fix didn't really solve the problem with ASAN,
      and also introduced issues with other CI runs.
      
      unrelated:
      - make runtest-cluster able to take multiple --single arguments
      d364ede5
    • Oran Agra's avatar
      update help.h before release (#10210) · ef931259
      Oran Agra authored
      ef931259
    • Ping Xie's avatar
      Fix cluster bus extensions backwards compatibility (#10206) · 8013af6f
      Ping Xie authored
      Before this commit, notused1 was incorrectly resized resulting with a clusterMsg that is not backwards compatible as expected.
      8013af6f
    • Moti Cohen's avatar
      Improve srand entropy (and fix Sentinel failures) (#10197) · 52b2fbe9
      Moti Cohen authored
      As Sentinel relies upon consensus algorithm, all sentinel instances,
      randomize a time to initiate their next attempt to become the
      leader of the group. But time after time, all raffled the same value.
      
      The problem is in the line `srand(time(NULL)^getpid())` such that
      all spinned up containers get same time (in seconds) and same pid
      which is always 1. Added material `tv_usec` and verify that even
      consecutive calls brings different values and makes the difference.
      52b2fbe9
    • Tobias Nießen's avatar
    • Harkrishn Patro's avatar
      Set default channel permission to resetchannels for 7.0 (#10181) · a43b6922
      Harkrishn Patro authored
      For backwards compatibility in 6.x, channels default permission was set to `allchannels` however with 7.0,
      we should modify it and the default value should be `resetchannels` for better security posture.
      Also, with selectors in ACL, a client doesn't have to set channel rules everytime and by default
      the value will be `resetchannels`.
      
      Before this change
      ```
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      127.0.0.1:6379>  acl setuser hp on nopass +@all ~*
      OK
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      2) "user hp on nopass ~* &* +@all"
      127.0.0.1:6379>  acl setuser hp1 on nopass -@all (%R~sales*)
      OK
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      2) "user hp on nopass ~* &* +@all"
      3) "user hp1 on nopass &* -@all (%R~sales* &* -@all)"
      ```
      
      After this change
      ```
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      127.0.0.1:6379> acl setuser hp on nopass +@all ~*
      OK
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      2) "user hp on nopass ~* resetchannels +@all"
      127.0.0.1:6379> acl setuser hp1 on nopass -@all (%R~sales*)
      OK
      127.0.0.1:6379> acl list
      1) "user default on nopass ~* &* +@all"
      2) "user hp on nopass ~* resetchannels +@all"
      3) "user hp1 on nopass resetchannels -@all (%R~sales* resetchannels -@all)"
      ```
      a43b6922
    • guybe7's avatar
      Add key-specs notes (#10193) · eedec155
      guybe7 authored
      Add optional `notes` to keyspecs.
      
      Other changes:
      
      1. Remove the "incomplete" flag from SORT and SORT_RO: it is misleading since "incomplete" means "this spec may not return all the keys it describes" but SORT and SORT_RO's specs (except the input key) do not return any keys at all.
      So basically:
      If a spec's begin_search is "unknown" you should not use it at all, you must use COMMAND KEYS;
      if a spec itself is "incomplete", you can use it to get a partial list of keys, but if you want all of them you must use COMMAND GETKEYS;
      otherwise, the spec will return all the keys
      
      2. `getKeysUsingKeySpecs` handles incomplete specs internally
      eedec155
    • Oran Agra's avatar
      fix cluster rebalance test race (#10207) · be0d2933
      Oran Agra authored
      Try to fix the rebalance cluster test that's failing with ASAN daily:
      
      Looks like `redis-cli --cluster rebalance` gets `ERR Please use SETSLOT only with masters` in `clusterManagerMoveSlot()`.
      it happens when `12-replica-migration-2.tcl` is run with ASAN in GH Actions.
      in `Resharding all the master #0 slots away from it`
      
      So the fix (assuming i got it right) is to call `redis-cli --cluster check` before `--cluster rebalance`.
      p.s. it looks like a few other checks in these tests needed that wait, added them too.
      
      Other changes:
      * in instances.tcl, make sure to catch tcl test crashes and let the rest of the code proceed, so that if there was
        a redis crash, we'll find it and print it too.
      * redis-cli, try to make sure it prints an error instead of silently exiting.
      
      specifically about redis-cli:
      1. clusterManagerMoveSlot used to print an error, only if the caller also asked for it (should be the other way around).
      2. clusterManagerCommandReshard asked for an error, but didn't use it (probably tried to avoid the double print).
      3. clusterManagerCommandRebalance didn't ask for the error, now it does.
      4. making sure that other places in clusterManagerCommandRebalance print something before exiting with an error.
      be0d2933