1. 30 May, 2022 1 commit
  2. 30 Jan, 2022 1 commit
    • guybe7's avatar
      Add key-specs notes (#10193) · eedec155
      guybe7 authored
      Add optional `notes` to keyspecs.
      
      Other changes:
      
      1. Remove the "incomplete" flag from SORT and SORT_RO: it is misleading since "incomplete" means "this spec may not return all the keys it describes" but SORT and SORT_RO's specs (except the input key) do not return any keys at all.
      So basically:
      If a spec's begin_search is "unknown" you should not use it at all, you must use COMMAND KEYS;
      if a spec itself is "incomplete", you can use it to get a partial list of keys, but if you want all of them you must use COMMAND GETKEYS;
      otherwise, the spec will return all the keys
      
      2. `getKeysUsingKeySpecs` handles incomplete specs internally
      eedec155
  3. 24 Jan, 2022 1 commit
    • yoav-steinberg's avatar
      Support function flags in script EVAL via shebang header (#10126) · 7eadc5ee
      yoav-steinberg authored
      In #10025 we added a mechanism for flagging certain properties for Redis Functions.
      This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 
      
      One good reason for this, other than the added functionality is because it addresses the
      poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
      during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
      Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
      unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.
      
      This PR:
      - Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
      - Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
        additional script annotations and even engine selection (instead of just lua) for scripts.
      - Provides backwards compatibility so scripts without the new annotations will behave as they did before.
      - Appropriate tests.
      - Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
        scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
        return the `MASTERDOWN` error as before.
      - Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
        compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
        which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
        won't work unless the script is explicitly marked as `allow-stale`.
      
      Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
      tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
      this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
      input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
      first line with a single line feed character.
      7eadc5ee
  4. 18 Jan, 2022 1 commit
    • Oran Agra's avatar
      New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122) · eef9c6b0
      Oran Agra authored
      The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
      explicit flags rather than just READ and WRITE. See discussion in #10040
      
      This PR defines two groups of flags:
      One about how redis internally handles the key (mutually-exclusive).
      The other is about the logical operation done from the user's point of view (3 mutually exclusive
      write flags, and one read flag, all optional).
      In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
      insert-only, we flag it as `RW` or `UPDATE`.
      here's the definition from the code:
      ```
      /* Key-spec flags *
       * -------------- */
      /* The following refer what the command actually does with the value or metadata
       * of the key, and not necessarily the user data or how it affects it.
       * Each key-spec may must have exaclty one of these. Any operation that's not
       * distinctly deletion, overwrite or read-only would be marked as RW. */
      #define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                        * doesn't necessarily returns it. */
      #define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                        * value of the key or its metadata. */
      #define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                        * the value of the key. */
      #define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
      /* The follwing refer to user data inside the value of the key, not the metadata
       * like LRU, type, cardinality. It refers to the logical operation on the user's
       * data (actual input strings / TTL), being used / returned / copied / changed,
       * It doesn't refer to modification or returning of metadata (like type, count,
       * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
       * Each key-spec may have one of the writes with or without access, or none: */
      #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                        * the value of the key. */
      #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                        * depend on the old value. */
      #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                        * modification or deletion of existing data. */
      #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                        * from the value of the key. */
      ```
      
      Unrelated changes:
      - generate-command-code.py is only compatible with python3 (modified the shabang)
      - generate-command-code.py print file on json parsing error
      - rename `shard_channel` key-spec flag to just `channel`.
      - add INCOMPLETE flag in input spec of SORT and SORT_RO
      eef9c6b0
  5. 15 Dec, 2021 1 commit
    • guybe7's avatar
      Auto-generate the command table from JSON files (#9656) · 86781600
      guybe7 authored
      Delete the hardcoded command table and replace it with an auto-generated table, based
      on a JSON file that describes the commands (each command must have a JSON file).
      
      These JSON files are the SSOT of everything there is to know about Redis commands,
      and it is reflected fully in COMMAND INFO.
      
      These JSON files are used to generate commands.c (using a python script), which is then
      committed to the repo and compiled.
      
      The purpose is:
      * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic.
      * drop the dependency between Redis-user and the commands.json in redis-doc.
      * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be
        done in a separate PR)
      * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release
        artifacts should be a large JSON, containing all the information about all of the commands, which will be
        generated from COMMAND's reply)
      * the byproduct of this is:
        * module commands will be able to provide that info and possibly be more of a first-class citizens
        * in theory, one may be able to generate a redis client library for a strictly typed language, by using this info.
      
      ### Interface changes
      
      #### COMMAND INFO's reply change (and arg-less COMMAND)
      
      Before this commit the reply at index 7 contained the key-specs list
      and reply at index 8 contained the sub-commands list (Both unreleased).
      Now, reply at index 7 is a map of:
      - summary - short command description
      - since - debut version
      - group - command group
      - complexity - complexity string
      - doc-flags - flags used for documentation (e.g. "deprecated")
      - deprecated-since - if deprecated, from which version?
      - replaced-by - if deprecated, which command replaced it?
      - history - a list of (version, what-changed) tuples
      - hints - a list of strings, meant to provide hints for clients/proxies. see https://github.com/redis/redis/issues/9876
      - arguments - an array of arguments. each element is a map, with the possibility of nesting (sub-arguments)
      - key-specs - an array of keys specs (already in unstable, just changed location)
      - subcommands - a list of sub-commands (already in unstable, just changed location)
      - reply-schema - will be added in the future (see https://github.com/redis/redis/issues/9845)
      
      more details on these can be found in https://github.com/redis/redis-doc/pull/1697
      
      only the first three fields are mandatory 
      
      #### API changes (unreleased API obviously)
      
      now they take RedisModuleCommand opaque pointer instead of looking up the command by name
      
      - RM_CreateSubcommand
      - RM_AddCommandKeySpec
      - RM_SetCommandKeySpecBeginSearchIndex
      - RM_SetCommandKeySpecBeginSearchKeyword
      - RM_SetCommandKeySpecFindKeysRange
      - RM_SetCommandKeySpecFindKeysKeynum
      
      Currently, we did not add module API to provide additional information about their commands because
      we couldn't agree on how the API should look like, see https://github.com/redis/redis/issues/9944
      
      .
      
      ### Somehow related changes
      1. Literals should be in uppercase while placeholder in lowercase. Now all the GEO* command
         will be documented with M|KM|FT|MI and can take both lowercase and uppercase
      
      ### Unrelated changes
      1. Bugfix: no_madaory_keys was absent in COMMAND's reply
      2. expose CMD_MODULE as "module" via COMMAND
      3. have a dedicated uint64 for ACL categories (instead of having them in the same uint64 as command flags)
      Co-authored-by: default avatarItamar Haber <itamar@garantiadata.com>
      86781600