1. 02 Jan, 2022 2 commits
    • Viktor Söderqvist's avatar
      Wait for replicas when shutting down (#9872) · 45a155bd
      Viktor Söderqvist authored
      
      
      To avoid data loss, this commit adds a grace period for lagging replicas to
      catch up the replication offset.
      
      Done:
      
      * Wait for replicas when shutdown is triggered by SIGTERM and SIGINT.
      
      * Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new
        blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients
        to call SHUTDOWN in parallel.
        Note that they don't expect a response unless an error happens and shutdown is aborted.
      
      * Log warning for each replica lagging behind when finishing shutdown.
      
      * CLIENT_PAUSE_WRITE while waiting for replicas.
      
      * Configurable grace period 'shutdown-timeout' in seconds (default 10).
      
      * New flags for the SHUTDOWN command:
      
          - NOW disables the grace period for lagging replicas.
      
          - FORCE ignores errors writing the RDB or AOF files which would normally
            prevent a shutdown.
      
          - ABORT cancels ongoing shutdown. Can't be combined with other flags.
      
      * New field in the output of the INFO command: 'shutdown_in_milliseconds'. The
        value is the remaining maximum time to wait for lagging replicas before
        finishing the shutdown. This field is present in the Server section **only**
        during shutdown.
      
      Not directly related:
      
      * When shutting down, if there is an AOF saving child, it is killed **even** if AOF
        is disabled. This can happen if BGREWRITEAOF is used when AOF is off.
      
      * Client pause now has end time and type (WRITE or ALL) per purpose. The
        different pause purposes are *CLIENT PAUSE command*, *failover* and
        *shutdown*. If clients are unpaused for one purpose, it doesn't affect client
        pause for other purposes. For example, the CLIENT UNPAUSE command doesn't
        affect client pause initiated by the failover or shutdown procedures. A completed
        failover or a failed shutdown doesn't unpause clients paused by the CLIENT
        PAUSE command.
      
      Notes:
      
      * DEBUG RESTART doesn't wait for replicas.
      
      * We already have a warning logged when a replica disconnects. This means that
        if any replica connection is lost during the shutdown, it is either logged as
        disconnected or as lagging at the time of exit.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      45a155bd
    • yoav-steinberg's avatar
      Generate RDB with Functions only via redis-cli --functions-rdb (#9968) · 1bf6d6f1
      yoav-steinberg authored
      
      
      This is needed in order to ease the deployment of functions for ephemeral cases, where user
      needs to spin up a server with functions pre-loaded.
      
      #### Details:
      
      * Added `--functions-rdb` option to _redis-cli_.
      * Functions only rdb via `REPLCONF rdb-filter-only functions`. This is a placeholder for a space
        separated inclusion filter for the RDB. In the future can be `REPLCONF rdb-filter-only
        "functions db:3 key-patten:user*"` and a complementing `rdb-filter-exclude` `REPLCONF`
        can also be added.
      * Handle "slave requirements" specification to RDB saving code so we can use the same RDB
        when different slaves express the same requirements (like functions-only) and not share the
        RDB when their requirements differ. This is currently just a flags `int`, but can be extended to
        a more complex structure with various filter fields.
      * make sure to support filters only in diskless replication mode (not to override the persistence file),
        we do that by forcing diskless (even if disabled by config)
      
      other changes:
      * some refactoring in rdb.c (extract portion of a big function to a sub-function)
      * rdb_key_save_delay used in AOFRW too
      * sendChildInfo takes the number of updated keys (incremental, rather than absolute)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1bf6d6f1
  2. 31 Dec, 2021 2 commits
  3. 30 Dec, 2021 3 commits
    • Viktor Söderqvist's avatar
      Modules: Mark all APIs non-experimental (#9983) · e4b3a257
      Viktor Söderqvist authored
      These exist for quite some time, and are no longer experimental
      e4b3a257
    • Binbin's avatar
      redis-cli: Add -X option and extend --cluster call take arg from stdin (#9980) · 4836ae32
      Binbin authored
      There are two changes in this commit:
      
      1. Add -X option to redis-cli.
      Currently `-x` can only be used to provide the last argument,
      so you can do `redis-cli dump keyname > key.dump`,
      and then do `redis-cli -x restore keyname 0 < key.dump`.
      
      But what if you want to add the replace argument (which comes last?).
      oran suggested adding such usage:
      `redis-cli -X <tag> restore keyname <tag> replace < key.dump`
      
      i.e. you're able to provide a string in the arguments that's gonna be
      substituted with the content from stdin.
      
      Note that the tag name should not conflict with others non-replaced args.
      And the -x and -X options are conflicting.
      
      Some usages:
      ```
      [root]# echo mypasswd | src/redis-cli -X passwd_tag mset username myname password passwd_tag                                                   OK
      [root]# echo username > username.txt
      [root]# head -c -1 username.txt | src/redis-cli -X name_tag mget name_tag password
      1) "myname"
      2) "mypasswd\n"
      ```
      
      2. Handle the combination of both `-x` and `--cluster` or `-X` and `--cluster`
      Extend the broadcast option to receive the last arg or <tag> arg from the stdin.
      
      Now we can use `redis-cli -x --cluster call <host>:<port> cmd`,
      or `redis-cli -X <tag> --cluster call <host>:<port> cmd <tag>`.
      (support part of #9899)
      4836ae32
    • Viktor Söderqvist's avatar
      Module API doc formatting from #9656 (#10026) · 5006eab5
      Viktor Söderqvist authored
      Add blank before lists, so that they will be rendered as lists.
      
      Commands RM_GetCommand and RM_CreateSubcommand were added in #9656.
      5006eab5
  4. 29 Dec, 2021 1 commit
    • Itamar Haber's avatar
      Add missing metadata to the commands SSOT files. (#10016) · aec8c577
      Itamar Haber authored
      Add missing information about commands, mainly from reviewing redis-doc and removing
      the metadata from it (https://github.com/redis/redis-doc/pull/1722)
      
      * Reintroduces CLUSTER S****S (supported by Redis) but missing from the JSON / docs (related? #9675).
        Note that without that json file, the command won't work (breaking change)
      * Adds the `replicas` argument (exists in Redis) to `CLIENT KILL`.
      * Adds `history` entries to several commands based on redis-doc's man pages.
      * Adds `since` to applicable command arguments based on `history` (this basically makes
        some of `history` redundant - perhaps at a later stage).
      * Uses proper semantic versioning in all version references.
      * Also removes `geoencodeCommand` and `geodecodeCommand` header
        declarations per b96af595.
      aec8c577
  5. 28 Dec, 2021 3 commits
  6. 27 Dec, 2021 1 commit
    • Itamar Haber's avatar
      Adds utils/gen-commands-json.py (#9958) · f810510b
      Itamar Haber authored
      Following #9656, this script generates a "commands.json" file from the output
      of the new COMMAND. The output of this script is used in redis/redis-doc#1714
      and by redis/redis-io#259. This also converts a couple of rogue dashes (in 
      'key-specs' and 'multiple-token' flags) to underscores (continues #9959).
      f810510b
  7. 26 Dec, 2021 3 commits
    • Binbin's avatar
      santize dump payload: fix carsh when zset with NAN score (#10002) · e84ccc3f
      Binbin authored
      `zslInsert` with a NAN score will crash the server.
      This one found by the `corrupt-dump-fuzzer`.
      e84ccc3f
    • Meir Shpilraien (Spielrein)'s avatar
      Add FUNCTION DUMP and RESTORE. (#9938) · 365cbf46
      Meir Shpilraien (Spielrein) authored
      Follow the conclusions to support Functions in redis cluster (#9899)
      
      Added 2 new FUNCTION sub-commands:
      1. `FUNCTION DUMP` - dump a binary payload representation of all the functions.
      2. `FUNCTION RESTORE <PAYLOAD> [FLUSH|APPEND|REPLACE]` - give the binary payload extracted
         using `FUNCTION DUMP`, restore all the functions on the given payload. Restore policy can be given to
         control how to handle existing functions (default is APPEND):
         * FLUSH: delete all existing functions.
         * APPEND: appends the restored functions to the existing functions. On collision, abort.
         * REPLACE: appends the restored functions to the existing functions. On collision,
           replace the old function with the new function.
      
      Modify `redis-cli --cluster add-node` to use `FUNCTION DUMP` to get existing functions from
      one of the nodes in the cluster, and `FUNCTION RESTORE` to load the same set of functions
      to the new node. `redis-cli` will execute this step before sending the `CLUSTER MEET` command
      to the new node. If `FUNCTION DUMP` returns an error, assume the current Redis version do not
      support functions and skip `FUNCTION RESTORE`. If `FUNCTION RESTORE` fails, abort and do not send
      the `CLUSTER MEET` command. If the new node already contains functions (before the `FUNCTION RESTORE`
      is sent), abort and do not add the node to the cluster. Test was added to verify
      `redis-cli --cluster add-node` works as expected. 
      365cbf46
    • Meir Shpilraien (Spielrein)'s avatar
      Changed fuction name to be case insensitive. (#9984) · 08ff606b
      Meir Shpilraien (Spielrein) authored
      Use case insensitive string comparison for function names (like we do for commands and configs)
      In addition, add verification that the functions only use the following characters: [a-zA-Z0-9_]
      08ff606b
  8. 23 Dec, 2021 1 commit
    • Yuta Hongo's avatar
      redis-cli: Add OUTPUT_JSON format type (#9954) · 63f606d3
      Yuta Hongo authored
      Introduce `redis-cli --json` option.
      CSV doesn't support Map type, then parsing SLOWLOG or HMSET with multiple args are not helpful.
      
      By default the `--json` implies RESP3, which makes it much more useful, and a `-2`
      option was added to force RESP2.
      When `HELLO 3` fails, it prints a warning message (which can be silenced with `-2`).
      If a user passed `-3` explicitly, the non-interactive mode will also exit with error without
      running the command, while in interactive session it'll keep running after printing the warning.
      
      JSON output would be helpful to parse Redis replies with other tools like jq.
      
      ```
      redis-cli --json slowlog get | jq
      
      [
        [
          1,
          1639677545,
          322362,
          [
            "HMSET",
            "dummy-key",
            "field1",
            "123,456,789... (152 more bytes)",
            "field2",
            "111,222,333... (140 more bytes)",
            "field3",
            "... (349 more arguments)"
          ],
          "127.0.0.1:49312",
          ""
        ]
      ]
      
      ```
      63f606d3
  9. 22 Dec, 2021 3 commits
    • guybe7's avatar
      Sort out mess around propagation and MULTI/EXEC (#9890) · 7ac21307
      guybe7 authored
      The mess:
      Some parts use alsoPropagate for late propagation, others using an immediate one (propagate()),
      causing edge cases, ugly/hacky code, and the tendency for bugs
      
      The basic idea is that all commands are propagated via alsoPropagate (i.e. added to a list) and the
      top-most call() is responsible for going over that list and actually propagating them (and wrapping
      them in MULTI/EXEC if there's more than one command). This is done in the new function,
      propagatePendingCommands.
      
      Callers to propagatePendingCommands:
      1. top-most call() (we want all nested call()s to add to the also_propagate array and just the top-most
         one to propagate them) - via `afterCommand`
      2. handleClientsBlockedOnKeys: it is out of call() context and it may propagate stuff - via `afterCommand`. 
      3. handleClientsBlockedOnKeys edge case: if the looked-up key is already expired, we will propagate the
         expire but will not unblock any client so `afterCommand` isn't called. in that case, we have to propagate
         the deletion explicitly.
      4. cron stuff: active-expire and eviction may also propagate stuff
      5. modules: the module API allows to propagate stuff from just about anywhere (timers, keyspace notifications,
         threads). I could have tried to catch all the out-of-call-context places but it seemed easier to handle it in one
         place: when we free the context. in the spirit of what was done in call(), only the top-most freeing of a module
         context may cause propagation.
      6. modules: when using a thread-safe ctx it's not clear when/if the ctx will be freed. we do know that the module
         must lock the GIL before calling RM_Replicate/RM_Call so we propagate the pending commands when
         releasing the GIL.
      
      A "known limitation", which were actually a bug, was fixed because of this commit (see propagate.tcl):
         When using a mix of RM_Call with `!` and RM_Replicate, the command would propagate out-of-order:
         first all the commands from RM_Call, and then the ones from RM_Replicate
      
      Another thing worth mentioning is that if, in the past, a client would issue a MULTI/EXEC with just one
      write command the server would blindly propagate the MULTI/EXEC too, even though it's redundant.
      not anymore.
      
      This commit renames propagate() to propagateNow() in order to cause conflicts in pending PRs.
      propagatePendingCommands is the only caller of propagateNow, which is now a static, internal helper function.
      
      Optimizations:
      1. alsoPropagate will not add stuff to also_propagate if there's no AOF and replicas
      2. alsoPropagate reallocs also_propagagte exponentially, to save calls to memmove
      
      Bugfixes:
      1. CONFIG SET can create evictions, sending notifications which can cause to dirty++ with modules.
         we need to prevent it from propagating to AOF/replicas
      2. We need to set current_client in RM_Call. buggy scenario:
         - CONFIG SET maxmemory, eviction notifications, module hook calls RM_Call
         - assertion in lookupKey crashes, because current_client has CONFIG SET, which isn't CMD_WRITE
      3. minor: in eviction, call propagateDeletion after notification, like active-expire and all commands
         (we always send a notification before propagating the command)
      7ac21307
    • Oran Agra's avatar
      Allow most CONFIG SET during loading, block some commands in async-loading (#9878) · 41e6e05d
      Oran Agra authored
      ## background
      Till now CONFIG SET was blocked during loading.
      (In the not so distant past, GET was disallowed too)
      
      We recently (not released yet) added an async-loading mode, see #9323,
      and during that time it'll serve CONFIG SET and any other command.
      And now we realized (#9770) that some configs, and commands are dangerous
      during async-loading.
      
      ## changes
      * Allow most CONFIG SET during loading (both on async-loading and normal loading)
      * Allow CONFIG REWRITE and CONFIG RESETSTAT during loading
      * Block a few config during loading (`appendonly`, `repl-diskless-load`, and `dir`)
      * Block a few commands during loading (list below)
      
      ## the blocked commands:
      * SAVE - obviously we don't wanna start a foregreound save during loading 8-)
      * BGSAVE - we don't mind to schedule one, but we don't wanna fork now
      * BGREWRITEAOF - we don't mind to schedule one, but we don't wanna fork now
      * MODULE - we obviously don't wanna unload a module during replication / rdb loading
        (MODULE HELP and MODULE LIST are not blocked)
      * SYNC / PSYNC - we're in the middle of RDB loading from master, must not allow sync
        requests now.
      * REPLICAOF / SLAVEOF - we're in the middle of replicating, maybe it makes sense to let
        the user abort it, but he couldn't do that so far, i don't wanna take any risk of bugs due to odd state.
      * CLUSTER - only allow [HELP, SLOTS, NODES, INFO, MYID, LINKS, KEYSLOT, COUNTKEYSINSLOT,
        GETKEYSINSLOT, RESET, REPLICAS, COUNT_FAILURE_REPORTS], for others, preserve the status quo
      
      ## other fixes
      * processEventsWhileBlocked had an issue when being nested, this could happen with a busy script
        during async loading (new), but also in a busy script during AOF loading (old). this lead to a crash in
        the scenario described in #6988
      41e6e05d
    • Hongcai Ren's avatar
      There is mismach between function sdssplitlen() comments and implementation (#4909) · b28dbef5
      Hongcai Ren authored
      when count is 0, return NULL
      b28dbef5
  10. 21 Dec, 2021 3 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Change FUNCTION CREATE, DELETE and FLUSH to be WRITE commands instead of MAY_REPLICATE. (#9953) · 3bcf1084
      Meir Shpilraien (Spielrein) authored
      The issue with MAY_REPLICATE is that all automatic mechanisms to handle
      write commands will not work. This require have a special treatment for:
      * Not allow those commands to be executed on RO replica.
      * Allow those commands to be executed on RO replica from primary connection.
      * Allow those commands to be executed on the RO replica from AOF.
      
      By setting those commands as WRITE commands we are getting all those properties from Redis.
      Test was added to verify that those properties work as expected.
      
      In addition, rearrange when and where functions are flushed. Before this PR functions were
      flushed manually on `rdbLoadRio` and cleaned manually on failure. This contradicts the
      assumptions that functions are data and need to be created/deleted alongside with the
      data. A side effect of this, for example, `debug reload noflush` did not flush the data but
      did flush the functions, `debug loadaof` flush the data but not the functions.
      This PR move functions deletion into `emptyDb`. `emptyDb` (renamed to `emptyData`) will
      now accept an additional flag, `NOFUNCTIONS` which specifically indicate that we do not
      want to flush the functions (on all other cases, functions will be flushed). Used the new flag
      on FLUSHALL and FLUSHDB only! Tests were added to `debug reload` and `debug loadaof`
      to verify that functions behave the same as the data.
      
      Notice that because now functions will be deleted along side with the data we can not allow
      `CLUSTER RESET` to be called from within a function (it will cause the function to be released
      while running), this PR adds `NO_SCRIPT` flag to `CLUSTER RESET`  so it will not be possible
      to be called from within a function. The other cluster commands are allowed from within a
      function (there are use-cases that uses `GETKEYSINSLOT` to iterate over all the keys on a
      given slot). Tests was added to verify `CLUSTER RESET` is denied from within a script.
      
      Another small change on this PR is that `RDBFLAGS_ALLOW_DUP` is also applicable on functions.
      When loading functions, if this flag is set, we will replace old functions with new ones on collisions. 
      3bcf1084
    • yancz2000's avatar
      Add --version support for redis-check-rdb (#9971) · 23b61734
      yancz2000 authored
      Let's you see which version of redis this tool is part of (similarly to redis-cli and redis-benchmark)
      23b61734
    • zhugezy's avatar
      Remove EVAL script verbatim replication, propagation, and deterministic execution logic (#9812) · 1b0968df
      zhugezy authored
      
      
      # Background
      
      The main goal of this PR is to remove relevant logics on Lua script verbatim replication,
      only keeping effects replication logic, which has been set as default since Redis 5.0.
      As a result, Lua in Redis 7.0 would be acting the same as Redis 6.0 with default
      configuration from users' point of view.
      
      There are lots of reasons to remove verbatim replication.
      Antirez has listed some of the benefits in Issue #5292:
      
      >1. No longer need to explain to users side effects into scripts.
          They can do whatever they want.
      >2. No need for a cache about scripts that we sent or not to the slaves.
      >3. No need to sort the output of certain commands inside scripts
          (SMEMBERS and others): this both simplifies and gains speed.
      >4. No need to store scripts inside the RDB file in order to startup correctly.
      >5. No problems about evicting keys during the script execution.
      
      When looking back at Redis 5.0, antirez and core team decided to set the config
      `lua-replicate-commands yes` by default instead of removing verbatim replication
      directly, in case some bad situations happened. 3 years later now before Redis 7.0,
      it's time to remove it formally.
      
      # Changes
      
      - configuration for lua-replicate-commands removed
        - created config file stub for backward compatibility
      - Replication script cache removed
        - this is useless under script effects replication
        - relevant statistics also removed
      - script persistence in RDB files is also removed
      - Propagation of SCRIPT LOAD and SCRIPT FLUSH to replica / AOF removed
      - Deterministic execution logic in scripts removed (i.e. don't run write commands
        after random ones, and sorting output of commands with random order)
        - the flags indicating which commands have non-deterministic results are kept as hints to clients.
      - `redis.replicate_commands()` & `redis.set_repl()` changed
        - now `redis.replicate_commands()` does nothing and return an 1
        - ...and then `redis.set_repl()` can be issued before `redis.replicate_commands()` now
      - Relevant TCL cases adjusted
      - DEBUG lua-always-replicate-commands removed
      
      # Other changes
      - Fix a recent bug comparing CLIENT_ID_AOF to original_client->flags instead of id. (introduced in #9780)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1b0968df
  11. 20 Dec, 2021 1 commit
    • Binbin's avatar
      Fix recent daily CI test failures (#9966) · febc3f63
      Binbin authored
      Recent PRs have introduced some failures, this commit
      try to fix these CI failures. Here are the changes:
      
      1. Enable debug-command in sentinel test.
      ```
      Master reboot in very short time: ERR DEBUG command not allowed. If the
      enable-debug-command option is set to "local", you can run it from a
      local connection, otherwise you need to set this option in the
      configuration file, and then restart the server.
      ```
      
      2. Enable protected-config in sentinel test.
      ```
      SDOWN is triggered by misconfigured instance replying with errors: ERR
      CONFIG SET failed (possibly related to argument 'dir') - can't set
      protected config
      ```
      
      3. Enable debug-command in cluster test.
      ```
      Verify slaves consistency: ERR DEBUG command not allowed. If the
      enable-debug-command option is set to "local", you can run it from a
      local connection, otherwise you need to set this option in the
      configuration file, and then restart the server.
      ```
      
      4. quicklist fill should be signed int.
      The reason for the modification is to eliminate the warning.
      Modify `int fill: QL_FILL_BITS` to `signed int fill: QL_FILL_BITS`
      
      The first three were introduced at #9920 (same issue).
      And the last one was introduced at #9962.
      febc3f63
  12. 19 Dec, 2021 2 commits
    • Binbin's avatar
      Fix some nonsense came from LGTM (#9962) · 0fb1aa06
      Binbin authored
      1. Local variable 's' hides a parameter of the same name.
      ```
      int anetTcpAccept(char *err, int s, char *ip, size_t ip_len, int *port) {
          if ((fd = anetGenericAccept(err,s,(struct sockaddr*)&sa,&salen)) == ANET_ERR){}
      }
      ```
      Change the parameter name from `s` to `serversock`,
      also unified with the header file definition.
      
      2. Comparison is always false because i <= 48.
      ```
      for (i = 0; i < DICT_STATS_VECTLEN-1; i++) {  // i < 49
          (i == DICT_STATS_VECTLEN-1)?">= ":"",  // i == 49
      }
      ```
      `i == DICT_STATS_VECTLEN-1` always result false, it is a dead code.
      
      3. Empty block without comment.
      `else if (!strcasecmp(opt,"ch")) {}`, add a comment to avoid warnings.
      
      4. Bit field fill of type int should have explicitly unsigned integral, explicitly signed integral, or enumeration type.
      Modify `int fill: QL_FILL_BITS;` to `unsigned int fill: QL_FILL_BITS;`
      
      5. The result of this call to reconnectingRedisCommand is not checked for null, but 80% of calls to reconnectingRedisCommand check for null.
      Just a cleanup job like others.
      0fb1aa06
    • YaacovHazan's avatar
      Protected configs and sensitive commands (#9920) · ae2f5b7b
      YaacovHazan authored
      Block sensitive configs and commands by default.
      
      * `enable-protected-configs` - block modification of configs with the new `PROTECTED_CONFIG` flag.
         Currently we add this flag to `dbfilename`, and `dir` configs,
         all of which are non-mutable configs that can set a file redis will write to.
      * `enable-debug-command` - block the `DEBUG` command
      * `enable-module-command` - block the `MODULE` command
      
      These have a default value set to `no`, so that these features are not
      exposed by default to client connections, and can only be set by modifying the config file.
      
      Users can change each of these to either `yes` (allow all access), or `local` (allow access from
      local TCP connections and unix domain connections)
      
      Note that this is a **breaking change** (specifically the part about MODULE command being disabled by default).
      I.e. we don't consider DEBUG command being blocked as an issue (people shouldn't have been using it),
      and the few configs we protected are unlikely to have been set at runtime anyway.
      On the other hand, it's likely to assume some users who use modules, load them from the config file anyway.
      Note that's the whole point of this PR, for redis to be more secure by default and reduce the attack surface on
      innocent users, so secure defaults will necessarily mean a breaking change.
      ae2f5b7b
  13. 18 Dec, 2021 1 commit
  14. 17 Dec, 2021 1 commit
    • ny0312's avatar
      Introduce memory management on cluster link buffers (#9774) · 792afb44
      ny0312 authored
      Introduce memory management on cluster link buffers:
       * Introduce a new `cluster-link-sendbuf-limit` config that caps memory usage of cluster bus link send buffers.
       * Introduce a new `CLUSTER LINKS` command that displays current TCP links to/from peers.
       * Introduce a new `mem_cluster_links` field under `INFO` command output, which displays the overall memory usage by all current cluster links.
       * Introduce a new `total_cluster_links_buffer_limit_exceeded` field under `CLUSTER INFO` command output, which displays the accumulated count of cluster links freed due to `cluster-link-sendbuf-limit`.
      792afb44
  15. 16 Dec, 2021 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Add FUNCTION FLUSH command to flush all functions (#9936) · 687210f1
      Meir Shpilraien (Spielrein) authored
      Added `FUNCTION FLUSH` command. The new sub-command allows delete all the functions.
      An optional `[SYNC|ASYNC]` argument can be given to control whether or not to flush the
      functions synchronously or asynchronously. if not given the default flush mode is chosen by
      `lazyfree-lazy-user-flush` configuration values.
      
      Add the missing `functions.tcl` test to the list of tests that are executed in test_helper.tcl,
      and call FUNCTION FLUSH in between servers in external mode
      687210f1
    • guybe7's avatar
      Command table: Sorted subcommands (#9951) · ffbe36fc
      guybe7 authored
      
      
      Sort the sub-commands so that every time we execute the script it generates the exact same results.
      This will case less merge conflicts if two PRs edit different json files.
      
      also:
      * make the script agnostic to where it is executed (more flexible).
      * add documentation about commands.c and the json files in the readme.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ffbe36fc
    • yoav-steinberg's avatar
      Multiparam config get. (#9914) · 70ff26b4
      yoav-steinberg authored
      Support doing `CONFIG GET <x> <y> <z>`, each of them can also be
      a pattern with wildcards.
      
      This avoids duplicates in the result by looping over the configs and for
      each once checking all the patterns, once a match is found for a pattern
      we move on to the next config.
      70ff26b4
    • ranshid's avatar
      Throw error on too long unix domain socket file path (#9826) · 28b5a653
      ranshid authored
      
      
      * Fix too long unix domain socket file path
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      28b5a653
  16. 15 Dec, 2021 4 commits
    • guybe7's avatar
      Auto-generate the command table from JSON files (#9656) · 86781600
      guybe7 authored
      Delete the hardcoded command table and replace it with an auto-generated table, based
      on a JSON file that describes the commands (each command must have a JSON file).
      
      These JSON files are the SSOT of everything there is to know about Redis commands,
      and it is reflected fully in COMMAND INFO.
      
      These JSON files are used to generate commands.c (using a python script), which is then
      committed to the repo and compiled.
      
      The purpose is:
      * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic.
      * drop the dependency between Redis-user and the commands.json in redis-doc.
      * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be
        done in a separate PR)
      * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release
        artifacts should be a large JSON, containing all the information about all of the commands, which will be
        generated from COMMAND's reply)
      * the byproduct of this is:
        * module commands will be able to provide that info and possibly be more of a first-class citizens
        * in theory, one may be able to generate a redis client library for a strictly typed language, by using this info.
      
      ### Interface changes
      
      #### COMMAND INFO's reply change (and arg-less COMMAND)
      
      Before this commit the reply at index 7 contained the key-specs list
      and reply at index 8 contained the sub-commands list (Both unreleased).
      Now, reply at index 7 is a map of:
      - summary - short command description
      - since - debut version
      - group - command group
      - complexity - complexity string
      - doc-flags - flags used for documentation (e.g. "deprecated")
      - deprecated-since - if deprecated, from which version?
      - replaced-by - if deprecated, which command replaced it?
      - history - a list of (version, what-changed) tuples
      - hints - a list of strings, meant to provide hints for clients/proxies. see https://github.com/redis/redis/issues/9876
      - arguments - an array of arguments. each element is a map, with the possibility of nesting (sub-arguments)
      - key-specs - an array of keys specs (already in unstable, just changed location)
      - subcommands - a list of sub-commands (already in unstable, just changed location)
      - reply-schema - will be added in the future (see https://github.com/redis/redis/issues/9845)
      
      more details on these can be found in https://github.com/redis/redis-doc/pull/1697
      
      only the first three fields are mandatory 
      
      #### API changes (unreleased API obviously)
      
      now they take RedisModuleCommand opaque pointer instead of looking up the command by name
      
      - RM_CreateSubcommand
      - RM_AddCommandKeySpec
      - RM_SetCommandKeySpecBeginSearchIndex
      - RM_SetCommandKeySpecBeginSearchKeyword
      - RM_SetCommandKeySpecFindKeysRange
      - RM_SetCommandKeySpecFindKeysKeynum
      
      Currently, we did not add module API to provide additional information about their commands because
      we couldn't agree on how the API should look like, see https://github.com/redis/redis/issues/9944
      
      .
      
      ### Somehow related changes
      1. Literals should be in uppercase while placeholder in lowercase. Now all the GEO* command
         will be documented with M|KM|FT|MI and can take both lowercase and uppercase
      
      ### Unrelated changes
      1. Bugfix: no_madaory_keys was absent in COMMAND's reply
      2. expose CMD_MODULE as "module" via COMMAND
      3. have a dedicated uint64 for ACL categories (instead of having them in the same uint64 as command flags)
      Co-authored-by: default avatarItamar Haber <itamar@garantiadata.com>
      86781600
    • Meir Shpilraien (Spielrein)'s avatar
      Use `server.dirty++` instead if `forceCommandPropagation` on FUNCTION commands (#9945) · fbfdf513
      Meir Shpilraien (Spielrein) authored
      Functions are considered data, so changing a function should be counted
      as a data change and should affect the persistence policy. For example
      if we want to persist an RDB each minute if there was a single change,
      functions should be counted as such change. Using `forceCommandPropagation`
      will not give us the desired effect and so we must switch to `server.dirty++`
      fbfdf513
    • 丽媛自己动's avatar
    • Wen Hui's avatar
      Error message improvement for CONFIG SET command (#9924) · a09bc504
      Wen Hui authored
      When CONFIG SET fails, print the name of the config that failed.
      This is helpful since config set is now variadic.
      
      however, there are cases where several configs have the same apply
      function, and we can't be sure which one of them caused the failure.
      a09bc504
  17. 13 Dec, 2021 2 commits
    • yoav-steinberg's avatar
      Fix possible int overflow when hashing an sds. (#9916) · c7dc17fc
      yoav-steinberg authored
      This caused a crash when adding elements larger than 2GB to a set (same goes for hash keys). See #8455.
      
      Details:
      * The fix makes the dict hash functions receive a `size_t` instead of an `int`. In practice the dict hash functions
        call siphash which receives a `size_t` and the callers to the hash function pass a `size_t` to it so the fix is trivial.
      * The issue was recreated by attempting to add a >2gb value to a set. Appropriate tests were added where I create
        a set with large elements and check basic functionality on it (SADD, SCARD, SPOP, etc...).
      * When I added the tests I also refactored a bit all the tests code which is run under the `--large-memory` flag.
        This removed code duplication for the test framework's `write_big_bulk` and `write_big_bulk` code and also takes
        care of not allocating the test frameworks helper huge string used by these tests when not run under `--large-memory`.
      * I also added the _violoations.tcl_ unit tests to be part of the entire test suite and leaned up non relevant list related
        tests that were in there. This was done in this PR because most of the _violations_ tests are "large memory" tests.
      c7dc17fc
    • Madelyn Olson's avatar
  18. 09 Dec, 2021 1 commit
  19. 08 Dec, 2021 2 commits
    • sundb's avatar
      Santize dump payload: fix crash when stream with duplicate consumes (#9918) · 7f0fae94
      sundb authored
      When rdb creates a consumer without determining whether it exists in advance,
      it may return NULL and crash if it encounters corrupt data with duplicate consumers.
      7f0fae94
    • yoav-steinberg's avatar
      Hide hidden configs from `config get` patterns. (#9888) · 07b13260
      yoav-steinberg authored
      Added `HIDDEN_CONFIG` to hide debug / dev / testing configs from CONFIG GET
      when it is used with a wildcard.
      These are not documented in redis.conf so now CONFIG GET only works when they
      are explicitly specified.
      
      The current configs are: 
      ```
      key-load-delay
      loading-process-events-interval-bytes
      rdb-key-save-delay
      use-exit-on-panic
      watchdog-period
      ```
      07b13260