1. 29 Mar, 2023 1 commit
    • Itamar Haber's avatar
      Overhauls command summaries and man pages. (#11942) · 0c3b8b7e
      Itamar Haber authored
      This is an attempt to normalize/formalize command summaries.
      
      Main actions performed:
      
      * Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands.
      * Starts with "An internal command...", "A container command...", etc... when applicable.
      * Always uses periods.
      * Refrains from referring to other commands. If this is needed, backquotes should be used for command names.
      * Tries to be very clear about the data type when applicable.
      * Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed."
      * Prefers being terse over verbose.
      * Tries to be consistent.
      0c3b8b7e
  2. 16 Mar, 2023 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Support for RM_Call on blocking commands (#11568) · d0da0a6a
      Meir Shpilraien (Spielrein) authored
      Allow running blocking commands from within a module using `RM_Call`.
      
      Today, when `RM_Call` is used, the fake client that is used to run command
      is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command
      that it is not allowed to block the client and in case it needs to block, it must
      fallback to some alternative (either return error or perform some default behavior).
      For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block.
      
      All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including
      module commands). When the command invocation finished, Redis asserts that
      the client was not blocked.
      
      This PR introduces the ability to call blocking command using `RM_Call` by
      passing a callback that will be called when the client will get unblocked.
      In order to do that, the user must explicitly say that he allow to perform blocking
      command by passing a new format specifier argument, `K`, to the `RM_Call`
      function. This new flag will tell Redis that it is allow to run blocking command
      and block the client. In case the command got blocked, Redis will return a new
      type of call reply (`REDISMODULE_REPLY_PROMISE`). This call reply indicates
      that the command got blocked and the user can set the on_unblocked handler using
      `RM_CallReplyPromiseSetUnblockHandler`.
      
      When clients gets unblocked, it eventually reaches `processUnblockedClients` function.
      This is where we check if the client is a fake module client and if it is, we call the unblock
      callback instead of performing the usual unblock operations.
      
      **Notice**: `RM_CallReplyPromiseSetUnblockHandler` must be called atomically
      along side the command invocation (without releasing the Redis lock in between).
      In addition, unlike other CallReply types, the promise call reply must be released
      by the module when the Redis GIL is acquired.
      
      The module can abort the execution on the blocking command (if it was not yet
      executed) using `RM_CallReplyPromiseAbort`. the API will return `REDISMODULE_OK`
      on success and `REDISMODULE_ERR` if the operation is already executed.
      **Notice** that in case of misbehave module, Abort might finished successfully but the
      operation will not really be aborted. This can only happened if the module do not respect
      the disconnect callback of the blocked client. 
      For pure Redis commands this can not happened.
      
      ### Atomicity Guarantees
      
      The API promise that the unblock handler will run atomically as an execution unit.
      This means that all the operation performed on the unblock handler will be wrapped
      with a multi exec transaction when replicated to the replica and AOF.
      The API **do not** grantee any other atomicity properties such as when the unblock
      handler will be called. This gives us the flexibility to strengthen the grantees (or not)
      in the future if we will decide that we need a better guarantees.
      
      That said, the implementation **does** provide a better guarantees when performing
      pure Redis blocking command like `BLPOP`. In this case the unblock handler will run
      atomically with the operation that got unblocked (for example, in case of `BLPOP`, the
      unblock handler will run atomically with the `LPOP` operation that run when the command
      got unblocked). This is an implementation detail that might be change in the future and the
      module writer should not count on that.
      
      ### Calling blocking commands while running on script mode (`S`)
      
      `RM_Call` script mode (`S`) was introduced on #0372. It is used for usecases where the
      command that was invoked on `RM_Call` comes from a user input and we want to make
      sure the user will not run dangerous commands like `shutdown`. Some command, such
      as `BLPOP`, are marked with `NO_SCRIPT` flag, which means they will not be allowed on
      script mode. Those commands are marked with  `NO_SCRIPT` just because they are
      blocking commands and not because they are dangerous. Now that we can run blocking
      commands on RM_Call, there is no real reason not to allow such commands on script mode.
      
      The underline problem is that the `NO_SCRIPT` flag is abused to also mark some of the
      blocking commands (notice that those commands know not to block the client if it is not
      allowed to do so, and have a fallback logic to such cases. So even if those commands
      were not marked with `NO_SCRIPT` flag, it would not harm Redis, and today we can
      already run those commands within multi exec).
      
      In addition, not all blocking commands are marked with `NO_SCRIPT` flag, for example
      `blmpop` are not marked and can run from within a script.
      
      Those facts shows that there are some ambiguity about the meaning of the `NO_SCRIPT`
      flag, and its not fully clear where it should be use.
      
      The PR suggest that blocking commands should not be marked with `NO_SCRIPT` flag,
      those commands should handle `CLIENT_DENY_BLOCKING` flag and only block when
      it's safe (like they already does today). To achieve that, the PR removes the `NO_SCRIPT`
      flag from the following commands:
      * `blmove`
      * `blpop`
      * `brpop`
      * `brpoplpush`
      * `bzpopmax`
      * `bzpopmin`
      * `wait`
      
      This might be considered a breaking change as now, on scripts, instead of getting
      `command is not allowed from script` error, the user will get some fallback behavior
      base on the command implementation. That said, the change matches the behavior
      of scripts and multi exec with respect to those commands and allow running them on
      `RM_Call` even when script mode is used.
      
      ### Additional RedisModule API and changes
      
      * `RM_BlockClientSetPrivateData` - Set private data on the blocked client without the
        need to unblock the client. This allows up to set the promise CallReply as the private
        data of the blocked client and abort it if the client gets disconnected.
      * `RM_BlockClientGetPrivateData` - Return the current private data set on a blocked client.
        We need it so we will have access to this private data on the disconnect callback.
      * On RM_Call, the returned reply will be added to the auto memory context only if auto
        memory is enabled, this allows us to keep the call reply for longer time then the context
        lifetime and does not force an unneeded borrow relationship between the CallReply and
        the RedisModuleContext.
      d0da0a6a
  3. 11 Mar, 2023 1 commit
    • guybe7's avatar
      Add reply_schema to command json files (internal for now) (#10273) · 4ba47d2d
      guybe7 authored
      Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
      Since ironing the details of the reply schema of each and every command can take a long time, we
      would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
      Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
      
      ### Background
      In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
      
      ### Motivation
      1. Documentation. This is the primary goal.
      2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
        languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
      3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
        testsuite, see the "Testing" section)
      
      ### Schema...
      4ba47d2d
  4. 20 Jan, 2022 1 commit
    • guybe7's avatar
      Add command tips to COMMAND DOCS (#10104) · 10bbeb68
      guybe7 authored
      Adding command tips (see https://redis.io/topics/command-tips) to commands.
      
      Breaking changes:
      1. Removed the "random" and "sort_for_script" flags. They are now command tips.
      (this isn't affecting redis behavior since #9812, but could affect some client applications
      that's relying on COMMAND command flags)
      
      Summary of changes:
      1. add BLOCKING flag (new flag) for all commands that could block. The ACL category with
        the same name is now implicit.
      2. move RANDOM flag to a `nondeterministic_output` tip
      3. move SORT_FOR_SCRIPT flag to `nondeterministic_output_order` tip
      3. add REQUEST_POLICY and RESPONSE_POLICY where appropriate as documented in the tips
      4. deprecate (ignore) the `random` flag for RM_CreateCommand
      
      Other notes:
      1. Proxies need to send `RANDOMKEY` to all shards and then select one key randomly.
        The other option is to pick a random shard and transfer `RANDOMKEY `to it, but that scheme
        fails if this specific shard is empty
      2. Remove CMD_RANDOM from `XACK` (i.e. XACK does not have RANDOM_OUTPUT)
         It was added in 9e4fb96c, I guess by mistake.
         Also from `(P)EXPIRETIME` (new command, was flagged "random" by mistake).
      3. Add `nondeterministic_output` to `OBJECT ENCODING` (for the same reason `XTRIM` has it:
         the reply may differ depending on the internal representation in memory)
      4. RANDOM on `HGETALL` was wrong (there due to a limitation of the old script sorting logic), now
        it's `nondeterministic_output_order`
      5. Unrelated: Hide CMD_PROTECTED from COMMAND
      10bbeb68
  5. 18 Jan, 2022 1 commit
    • Oran Agra's avatar
      New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122) · eef9c6b0
      Oran Agra authored
      The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
      explicit flags rather than just READ and WRITE. See discussion in #10040
      
      This PR defines two groups of flags:
      One about how redis internally handles the key (mutually-exclusive).
      The other is about the logical operation done from the user's point of view (3 mutually exclusive
      write flags, and one read flag, all optional).
      In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
      insert-only, we flag it as `RW` or `UPDATE`.
      here's the definition from the code:
      ```
      /* Key-spec flags *
       * -------------- */
      /* The following refer what the command actually does with the value or metadata
       * of the key, and not necessarily the user data or how it affects it.
       * Each key-spec may must have exaclty one of these. Any operation that's not
       * distinctly deletion, overwrite or read-only would be marked as RW. */
      #define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                        * doesn't necessarily returns it. */
      #define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                        * value of the key or its metadata. */
      #define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                        * the value of the key. */
      #define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
      /* The follwing refer to user data inside the value of the key, not the metadata
       * like LRU, type, cardinality. It refers to the logical operation on the user's
       * data (actual input strings / TTL), being used / returned / copied / changed,
       * It doesn't refer to modification or returning of metadata (like type, count,
       * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
       * Each key-spec may have one of the writes with or without access, or none: */
      #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                        * the value of the key. */
      #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                        * depend on the old value. */
      #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                        * modification or deletion of existing data. */
      #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                        * from the value of the key. */
      ```
      
      Unrelated changes:
      - generate-command-code.py is only compatible with python3 (modified the shabang)
      - generate-command-code.py print file on json parsing error
      - rename `shard_channel` key-spec flag to just `channel`.
      - add INCOMPLETE flag in input spec of SORT and SORT_RO
      eef9c6b0
  6. 15 Dec, 2021 1 commit
    • guybe7's avatar
      Auto-generate the command table from JSON files (#9656) · 86781600
      guybe7 authored
      Delete the hardcoded command table and replace it with an auto-generated table, based
      on a JSON file that describes the commands (each command must have a JSON file).
      
      These JSON files are the SSOT of everything there is to know about Redis commands,
      and it is reflected fully in COMMAND INFO.
      
      These JSON files are used to generate commands.c (using a python script), which is then
      committed to the repo and compiled.
      
      The purpose is:
      * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic.
      * drop the dependency between Redis-user and the commands.json in redis-doc.
      * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be
        done in a separate PR)
      * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release
        artifacts should be a large JSON, containing all the information about all of the commands, ...
      86781600