1. 17 Feb, 2022 2 commits
    • yoav-steinberg's avatar
      aof rewrite and rdb save counters in info (#10178) · 56fa48ff
      yoav-steinberg authored
      Add aof_rewrites and rdb_snapshots counters to info.
      This is useful to figure our if a rewrite or snapshot happened since last check.
      This was part of the (ongoing) effort to provide a safe backup solution for multipart-aof backups.
      56fa48ff
    • chenyang8094's avatar
      Adapt redis-check-aof tool for Multi Part Aof (#10061) · a50aa29b
      chenyang8094 authored
      Modifications of this PR:
      1. Support the verification of `Multi Part AOF`, while still maintaining support for the
        old-style `AOF/RDB-preamble`. `redis-check-aof` will automatically choose which
        mode to use according to the incoming file format.
         
      `Usage: redis-check-aof [--fix|--truncate-to-timestamp $timestamp] <AOF/manifest>`
       
      2. Refactor part of the code to make it easier to understand
      3. Currently only supports truncate  (`--fix` or `--truncate-to-timestamp`) the last AOF
        file (may be `BASE` or `INCR`)
      
      The reasons for 3 above:
      - for `--fix`: Only the last AOF may be truncated, this is guaranteed by redis
      - for `--truncate-to-timestamp`:  Normally, we only have `BASE` + `INCR` files
        at most, and `BASE` cannot be truncated(It only contains a timestamp annotation
        at the beginning of the file), so only `INCR` can be truncated. If we have a
        `BASE+INCR1+INCR2` file (meaning we have an interrupted AOFRW), Only `INCR2`
        files can be truncated at this time. If we still insist on truncate `INCR1`, we need to
        manually delete `INCR2` and update the manifest file, then re-run `redis-check-aof`
      - If we want to support truncate any file, we need to add very complicated code to support
        the atomic modification of multiple file deletion and update manifest, I think this is unnecessary
      a50aa29b
  2. 16 Feb, 2022 2 commits
  3. 15 Feb, 2022 1 commit
    • Yossi Gottlieb's avatar
      Fix OpenSSL 3.0.x related issues. (#10291) · 3881f785
      Yossi Gottlieb authored
      * Drop obsolete initialization calls.
      * Use decoder API for DH parameters.
      * Enable auto DH parameters if not explicitly used, which should be the
        preferred configuration going forward.
      3881f785
  4. 14 Feb, 2022 1 commit
  5. 13 Feb, 2022 3 commits
    • Oran Agra's avatar
      Fix and improve module error reply statistics (#10278) · b099889a
      Oran Agra authored
      This PR handles several aspects
      1. Calls to RM_ReplyWithError from thread safe contexts don't violate thread safety.
      2. Errors returning from RM_Call to the module aren't counted in the statistics (they
        might be handled silently by the module)
      3. When a module propagates a reply it got from RM_Call to it's client, then the error
        statistics are counted.
      
      This is done by:
      1. When appending an error reply to the output buffer, we avoid updating the global
        error statistics, instead we cache that error in a deferred list in the client struct.
      2. When creating a RedisModuleCallReply object, the deferred error list is moved from
        the client into that object.
      3. when a module calls RM_ReplyWithCallReply we copy the deferred replies to the dest
        client (if that's a real client, then that's when the error statistics are updated to the server)
      
      Note about RM_ReplyWithCallReply: if the original reply had an array with errors, and the module
      replied with just a portion of the original reply, and not the entire reply, the errors are currently not
      propagated and the errors stats will not get propagated.
      
      Fix #10180
      b099889a
    • Oran Agra's avatar
      Add workflow_dispatch filters for daily CI. (#10289) · 1193e96d
      Oran Agra authored
      sometimes you just wanna run one test on one system (e.g. memefficiency
      on macos), so you want all other tests to be skipped
      1193e96d
    • Binbin's avatar
      Regression test for sync psync crash (#10288) · 62c8be28
      Binbin authored
      Added regression tests for #10020 / #10081 / #10243.
      The above PRs fixed some crashes due to an asserting,
      see function `clientHasPendingReplies` (introduced in #9166).
      
      This commit added some tests to cover the above scenario.
      These tests will all fail in #9166, althought fixed not,
      there is value in adding these tests to cover and verify
      the changes. And it also can cover #8868 (verify the logs).
      
      Other changes: 
      1. Reduces the wait time in `waitForBgsave` and `waitForBgrewriteaof`
      from 1s to 50ms, which should reduce the time for some tests.
      2. Improve the test infra to print context when `assert_match` fails.
      3. Improve the test infra to print `$error` when `assert_error` fails.
      ```
      Expected an error matching 'ERR*' but got 'OK' (context: type eval line 4 cmd {assert_error "ERR*" {r set a b}} proc ::test)
      ```
      62c8be28
  6. 11 Feb, 2022 4 commits
    • yoav-steinberg's avatar
      Fix Eval scripts defrag (broken 7.0 in RC1) (#10271) · 2eb9b196
      yoav-steinberg authored
      Remove scripts defragger since it was broken since #10126 (released in 7.0 RC1).
      would crash the server if defragger starts in a server that contains eval scripts.
      
      In #10126 the global `lua_script` dict became a dict to a custom `luaScript` struct with an internal `robj`
      in it instead of a generic `sds` -> `robj` dict. This means we need custom code to defrag it and since scripts
      should never really cause much fragmentation it makes more sense to simply remove the defrag code for scripts.
      2eb9b196
    • sundb's avatar
      Fix duplicate module options define (#10284) · 5f0119ca
      sundb authored
      
      
      The bug is introduced by #9323. (released in 7.0 RC1)
      The define of `REDISMODULE_OPTIONS_HANDLE_IO_ERRORS` and `REDISMODULE_OPTION_NO_IMPLICIT_SIGNAL_MODIFIED` have the same value.
      
      This will result in skipping `signalModifiedKey()` after `RM_CloseKey()` if the module has set
      `REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD` option.
      The implication is missing WATCH and client side tracking invalidations.
      
      Other changes:
      - add `no-implicit-signal-modified` to the options in INFO modules
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5f0119ca
    • chenyang8094's avatar
      Modify AOF preamble related logs, and change the RDB aux field (#10283) · a2f2b6f5
      chenyang8094 authored
      In multi-part aof,  We no longer have the concept of `RDB-preamble`, so the related logs should be removed.
      However, in order to print compatible logs when loading old-style AOFs, we also have to keep the relevant code.
      Additionally, when saving an RDB, change the RDB aux field from "aof-preamble" to "aof-base".
      a2f2b6f5
    • Harkrishn Patro's avatar
  7. 10 Feb, 2022 2 commits
  8. 09 Feb, 2022 4 commits
  9. 08 Feb, 2022 8 commits
    • Oran Agra's avatar
      Attempt to fix a rare crash in cluster tests. (#10265) · aa9beaca
      Oran Agra authored
      The theory is that a replica gets disconnected from within REPLCONF ACK,
      so when we go up the stack, we'll crash when attempting to access
      c->cmd->flags
      aa9beaca
    • gms's avatar
      Fix SENTINEL DEBUG with wrong arguments (#10258) · 0990dec3
      gms authored
      There are two issues in SENTINEL DEBUG:
      1. The error message should mention SENTINEL DEBUG
      2. Add missing reuturn in args parse.
      
      ```
      redis> sentinel debug INFO-PERIOD aaa
      (error) ERR Invalid argument 'aaa' for SENTINEL SET 'INFO-PERIOD'
      
      redis> sentinel debug a b c d
      (error) ERR Unknown option or number of arguments for SENTINEL SET 'a'
      redis> ping
      (error) ERR Unknown option or number of arguments for SENTINEL SET 'b'
      ```
      
      Introduced in #9291. Also do some cleanups in the code.
      0990dec3
    • weiguo's avatar
      Remove duplicate header file include (#10264) · 34c288fe
      weiguo authored
      34c288fe
    • Viktor Söderqvist's avatar
      Remove assert and refuse delete expired on ro replicas (#10248) · b571c960
      Viktor Söderqvist authored
      There's an assertion added recently to make sure that non-write commands don't use lookupKeyWrite,
      It was initially meant to be used only on read-only replicas, but we thought it'll not have enough coverage,
      so used it on the masters too.
      We now realize that in some cases this can cause issues for modules, so we remove the assert.
      
      Other than that, we also make sure not to force expireIfNeeded on read-only replicas.
      even if they somehow run a write command.
      
      See https://github.com/redis/redis/pull/9572#discussion_r800179373
      b571c960
    • Wen Hui's avatar
      Make INFO command variadic (#6891) · 2e1bc942
      Wen Hui authored
      
      
      This is an enhancement for INFO command, previously INFO only support one argument
      for different info section , if user want to get more categories information, either perform
      INFO all / default or calling INFO for multiple times.
      
      **Description of the feature**
      
      The goal of adding this feature is to let the user retrieve multiple categories via the INFO
      command, and still avoid emitting the same section twice.
      
      A use case for this is like Redis Sentinel, which periodically calling INFO command to refresh
      info from monitored Master/Slaves, only Server and Replication part categories are used for
      parsing information. If the INFO command can return just enough categories that client side
      needs, it can save a lot of time for client side parsing it as well as network bandwidth.
      
      **Implementation**
      To share code between redis, sentinel, and other users of INFO (DEBUG and modules),
      we have a new `genInfoSectionDict` function that returns a dict and some boolean flags
      (e.g. `all`) to the caller (built from user input).
      Sentinel is later purging unwanted sections from that, and then it is forwarded to the info `genRedisInfoString`.
      
      **Usage Examples**
      INFO Server Replication   
      INFO CPU Memory
      INFO default commandstats
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2e1bc942
    • yoav-steinberg's avatar
      Consistent erros returned from EVAL scripts (#10218) · b76016a9
      yoav-steinberg authored
      This PR handles inconsistencies in errors returned from lua scripts.
      Details of the problem can be found in #10165.
      
      ### Changes
      
      - Remove double stack trace. It's enough that a stack trace is automatically added by the engine's error handler
        see https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/function_lua.c#L472-L485
        and https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/eval.c#L243-L255
      - Make sure all errors a preceded with an error code. Passing a simple string to `luaPushError()` will prepend it
        with a generic `ERR` error code.
      - Make sure lua error table doesn't include a RESP `-` error status. Lua stores redis error's as a lua table with a
        single `err` field and a string. When the string is translated back to RESP we add a `-` to it.
        See https://github.com/redis/redis/blob/d0bc4fff18afdf9e5421cc88e23ffbb876ecaec3/src/script_lua.c#L510-L517
        So there's no need to store it in the lua table.
      
      ### Before & After
      ```diff
      --- <unnamed>
      +++ <unnamed>
      @@ -1,14 +1,14 @@
        1: config set maxmemory 1
        2: +OK
        3: eval "return redis.call('set','x','y')" 0
      - 4: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: @user_script: 1: -OOM command not allowed when used memory > 'maxmemory'.
      + 4: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: OOM command not allowed when used memory > 'maxmemory'.
        5: eval "return redis.pcall('set','x','y')" 0
      - 6: -@user_script: 1: -OOM command not allowed when used memory > 'maxmemory'.
      + 6: -OOM command not allowed when used memory > 'maxmemory'.
        7: eval "return redis.call('select',99)" 0
        8: -ERR Error running script (call to 4ad5abfc50bbccb484223905f9a16f09cd043ba8): @user_script:1: ERR DB index is out of range
        9: eval "return redis.pcall('select',99)" 0
       10: -ERR DB index is out of range
       11: eval_ro "return redis.call('set','x','y')" 0
      -12: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: @user_script: 1: Write commands are not allowed from read-only scripts.
      +12: -ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: ERR Write commands are not allowed from read-only scripts.
       13: eval_ro "return redis.pcall('set','x','y')" 0
      -14: -@user_script: 1: Write commands are not allowed from read-only scripts.
      +14: -ERR Write commands are not allowed from read-only scripts.
      ```
      b76016a9
    • guybe7's avatar
      X[AUTO]CLAIM should skip deleted entries (#10227) · 3c3e6cc1
      guybe7 authored
      Fix #7021 #8924 #10198
      
      # Intro
      Before this commit X[AUTO]CLAIM used to transfer deleted entries from one
      PEL to another, but reply with "nil" for every such entry (instead of the entry id).
      The idea (for XCLAIM) was that the caller could see this "nil", realize the entry
      no longer exists, and XACK it in order to remove it from PEL.
      The main problem with that approach is that it assumes there's a correlation
      between the index of the "id" arguments and the array indices, which there
      isn't (in case some of the input IDs to XCLAIM never existed/read):
      
      ```
      127.0.0.1:6379> XADD x 1 f1 v1
      "1-0"
      127.0.0.1:6379> XADD x 2 f1 v1
      "2-0"
      127.0.0.1:6379> XADD x 3 f1 v1
      "3-0"
      127.0.0.1:6379> XGROUP CREATE x grp 0
      OK
      127.0.0.1:6379> XREADGROUP GROUP grp Alice COUNT 2 STREAMS x >
      1) 1) "x"
         2) 1) 1) "1-0"
               2) 1) "f1"
                  2) "v1"
            2) 1) "2-0"
               2) 1) "f1"
                  2) "v1"
      127.0.0.1:6379> XDEL x 1 2
      (integer) 2
      127.0.0.1:6379> XCLAIM x grp Bob 0 0-99 1-0 1-99 2-0
      1) (nil)
      2) (nil)
      ```
      
      # Changes
      Now,  X[AUTO]CLAIM acts in the following way:
      1. If one tries to claim a deleted entry, we delete it from the PEL we found it in
        (and the group PEL too). So de facto, such entry is not claimed, just cleared
        from PEL (since anyway it doesn't exist in the stream)
      2. since we never claim deleted entries, X[AUTO]CLAIM will never return "nil"
        instead of an entry.
      3. add a new element to XAUTOCLAIM's response (see below)
      
      # Knowing which entries were cleared from the PEL
      The caller may want to log any entries that were found in a PEL but deleted from
      the stream itself (it would suggest that there might be a bug in the application:
      trimming the stream while some entries were still no processed by the consumers)
      
      ## XCLAIM
      the set {XCLAIM input ids} - {XCLAIM returned ids} contains all the entry ids that were
      not claimed which means they were deleted (assuming the input contains only entries
      from some PEL). The user doesn't need to XACK them because XCLAIM had already
      deleted them from the source PEL.
      
      ## XAUTOCLAIM
      XAUTOCLAIM has a new element added to its reply: it's an array of all the deleted
      stream IDs it stumbled upon.
      
      This is somewhat of a breaking change since X[AUTO]CLAIM used to be able to reply
      with "nil" and now it can't... But since it was undocumented (and generally a bad idea
      to rely on it, as explained above) the breakage is not that bad.
      3c3e6cc1
    • Oran Agra's avatar
      Handle key-spec flags with modules (#10237) · 66be30f7
      Oran Agra authored
      - add COMMAND GETKEYSANDFLAGS sub-command
      - add RM_KeyAtPosWithFlags and GetCommandKeysWithFlags
      - RM_KeyAtPos and RM_CreateCommand set flags requiring full access for keys
      - RM_CreateCommand set VARIABLE_FLAGS
      - expose `variable_flags` flag in COMMAND INFO key-specs
      - getKeysFromCommandWithSpecs prefers key-specs over getkeys-api
      - add tests for all of these
      66be30f7
  10. 07 Feb, 2022 7 commits
    • ivanstosic-janea's avatar
      Fix protocol error caused by redis-benchmark (#10236) · bb875603
      ivanstosic-janea authored
      The protocol error was caused by the buggy `writeHandler` in `redis-benchmark.c`,
      which didn't handle one of the cases, thereby repeating data, leading to protocol errors
      when the values being sent are very long.
      
      This PR fixes #10233, issue introduced by #7959
      bb875603
    • Avital-Fine's avatar
    • Binbin's avatar
      COMMAND DOCS avoid adding summary/since if they don't exist (#10252) · 7f4cca11
      Binbin authored
      If summary or since is empty, we used to return NULL in
      COMMAND DOCS. Currently all redis commands will have these
      two fields.
      
      But not for module command, summary and since are optional
      for RM_SetCommandInfo. With the change in #10043, if a module
      command doesn't have the summary or since, redis-cli will
      crash (see #10250).
      
      In this commit, COMMAND DOCS avoid adding summary or since
      when they are missing.
      7f4cca11
    • Binbin's avatar
      Fix redis-cli with sentinel crash due to SENTINEL DEBUG missing summary (#10250) · b95beeb5
      Binbin authored
      Fix redis-cli with sentinel crash due to SENTINEL DEBUG missing summary
      
      Because SENTINEL DEBUG missing summary in its json file,
      with the change in #10043, the following assertion will fail.
      ```
      [redis]# src/redis-cli -p 26379
      redis-cli: redis-cli.c:678: cliInitCommandHelpEntry: Assertion `reply->type == 1' failed.
      ```
      
      This commit add the summary and complexity for SENTINEL DEBUG,
      which introduced in #9291, and also improved the help message.
      b95beeb5
    • yoav-steinberg's avatar
      acl check api for functions and eval (#10220) · 9dfeda58
      yoav-steinberg authored
      Changes:
      1. Adds the `redis.acl_check_cmd()` api to lua scripts. It can be used to check if the
        current user has permissions to execute a given command. The new function receives
        the command to check as an argument exactly like `redis.call()` receives the command
        to execute as an argument.
      2. In the PR I unified the code used to convert lua arguments to redis argv arguments from
        both the new `redis.acl_check_cmd()` API and the `redis.[p]call()` API. This cleans up
        potential duplicate code.
      3. While doing the refactoring in 2 I noticed there's an optimization to reduce allocation calls
        when parsing lua arguments into an `argv` array in the `redis.[p]call()` implementation.
        These optimizations were introduced years ago in 48c49c48
        and 4f686555. It is unclear why this was added.
        The original commit message claims a 4% performance increase which I couldn't recreate
        and might not be worth it even if it did recreate. This PR removes that optimization.
        Following are details of the benchmark I did that couldn't reveal any performance
        improvements due to this optimization:
      
      ```
      benchmark 1: src/redis-benchmark -P 500 -n 10000000 eval 'return redis.call("ping")' 0
      benchmark 2: src/redis-benchmark -P 500 -r 1000 -n 1000000 eval 'return redis.call("mset","k1__rand_int__","v1__rand_int__","k2__rand_int__","v2__rand_int__","k3__rand_int__","v3__rand_int__","k4__rand_int__","v4__rand_int__")' 0
      benchmark 3: src/redis-benchmark -P 500 -r 1000 -n 100000 eval "for i=1,100,1 do redis.call('set','kk'..i,'vv'..__rand_int__) end return redis.call('get','kk5')" 0
      benchmark 4: src/redis-benchmark -P 500 -r 1000 -n 1000000 eval 'return redis.call("mset","k1__rand_int__","v1__rand_int__","k2__rand_int__","v2__rand_int__","k3__rand_int__","v3__rand_int__","k4__rand_int__","v4__rand_int__xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")'
      ```
      I ran the benchmark on this branch with and without commit 68b71680a4d3bb8f0509e06578a9f15d05b92a47
      Results in requests per second:
      cmd | without optimization | without optimization 2nd run | with original optimization | with original optimization 2nd run
      -- | -- | -- | -- | --
      1 | 461233.34 | 477395.31 | 471098.16 | 469946.91
      2 | 34774.14 | 35469.8 | 35149.38 | 34464.93
      3 | 6390.59 | 6281.41 | 6146.28 | 6464.12
      4 | 28005.71 |   | 27965.77 |  
      
      As you can see, different use cases showed identical or negligible performance differences.
      So finally I decided to chuck the original optimization and simplify the code.
      9dfeda58
    • Binbin's avatar
      create-cluster clean now will clean appendonlydir (#10223) · c5e3d135
      Binbin authored
      In #9788, now we stores all persistent append-only files in
      a dedicated directory. The name of the directory is determined
      by the appenddirname configuration parameter in redis.conf.
      Now each node have a separate folder.
      Update create-cluster clean to clean this default directory.
      c5e3d135
    • weiguo's avatar
      d6e9cde5
  11. 06 Feb, 2022 4 commits
  12. 05 Feb, 2022 2 commits
    • Oran Agra's avatar
      fix file permissions for scripts in utils folder (#10241) · c42b1633
      Oran Agra authored
      make sure the scripts are executable
      c42b1633
    • Jason Elbaum's avatar
      redis-cli generates command help tables from the results of COMMAND (#10043) · 5b17909c
      Jason Elbaum authored
      
      
      This is a followup to #9656 and implements the following step mentioned in that PR:
      
      * When possible, extract all the help and completion tips from COMMAND DOCS (Redis 7.0 and up)
      * If COMMAND DOCS fails, use the static help.h compiled into redis-cli.
      * Supplement additional command names from COMMAND (pre-Redis 7.0)
      
      The last step is needed to add module command and other non-standard commands.
      
      This PR does not change the interactive hinting mechanism, which still uses only the param
      strings to provide somewhat unreliable and inconsistent command hints (see #8084).
      That task is left for a future PR. 
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5b17909c