1. 24 Jan, 2022 1 commit
    • yoav-steinberg's avatar
      Support function flags in script EVAL via shebang header (#10126) · 7eadc5ee
      yoav-steinberg authored
      In #10025 we added a mechanism for flagging certain properties for Redis Functions.
      This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 
      
      One good reason for this, other than the added functionality is because it addresses the
      poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
      during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
      Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
      unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.
      
      This PR:
      - Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
      - Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
        additional script annotations and even engine selection (instead of just lua) for scripts.
      - Provides backwards compatibility so scripts without the new annotations will behave as they did before.
      - Appropriate tests.
      - Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
        scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
        return the `MASTERDOWN` error as before.
      - Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
        compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
        which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
        won't work unless the script is explicitly marked as `allow-stale`.
      
      Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
      tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
      this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
      input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
      first line with a single line feed character.
      7eadc5ee
  2. 18 Jan, 2022 1 commit
    • Oran Agra's avatar
      New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122) · eef9c6b0
      Oran Agra authored
      The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
      explicit flags rather than just READ and WRITE. See discussion in #10040
      
      This PR defines two groups of flags:
      One about how redis internally handles the key (mutually-exclusive).
      The other is about the logical operation done from the user's point of view (3 mutually exclusive
      write flags, and one read flag, all optional).
      In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
      insert-only, we flag it as `RW` or `UPDATE`.
      here's the definition from the code:
      ```
      /* Key-spec flags *
       * -------------- */
      /* The following refer what the command actually does with the value or metadata
       * of the key, and not necessarily the user data or how it affects it.
       * Each key-spec may must have exaclty one of these. Any operation that's not
       * distinctly deletion, overwrite or read-only would be marked as RW. */
      #define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                        * doesn't necessarily returns it. */
      #define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                        * value of the key or its metadata. */
      #define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                        * the value of the key. */
      #define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
      /* The follwing refer to user data inside the value of the key, not the metadata
       * like LRU, type, cardinality. It refers to the logical operation on the user's
       * data (actual input strings / TTL), being used / returned / copied / changed,
       * It doesn't refer to modification or returning of metadata (like type, count,
       * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
       * Each key-spec may have one of the writes with or without access, or none: */
      #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                        * the value of the key. */
      #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                        * depend on the old value. */
      #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                        * modification or deletion of existing data. */
      #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                        * from the value of the key. */
      ```
      
      Unrelated changes:
      - generate-command-code.py is only compatible with python3 (modified the shabang)
      - generate-command-code.py print file on json parsing error
      - rename `shard_channel` key-spec flag to just `channel`.
      - add INCOMPLETE flag in input spec of SORT and SORT_RO
      eef9c6b0
  3. 15 Dec, 2021 1 commit
    • guybe7's avatar
      Auto-generate the command table from JSON files (#9656) · 86781600
      guybe7 authored
      Delete the hardcoded command table and replace it with an auto-generated table, based
      on a JSON file that describes the commands (each command must have a JSON file).
      
      These JSON files are the SSOT of everything there is to know about Redis commands,
      and it is reflected fully in COMMAND INFO.
      
      These JSON files are used to generate commands.c (using a python script), which is then
      committed to the repo and compiled.
      
      The purpose is:
      * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic.
      * drop the dependency between Redis-user and the commands.json in redis-doc.
      * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be
        done in a separate PR)
      * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release
        artifacts should be a large JSON, containing all the information about all of the commands, ...
      86781600