1. 14 Jun, 2024 1 commit
    • Jo's avatar
      Update `FIELDS` argument to block type for HFE commands schema (#13339) · 871c9859
      Jo authored
      I reviewed `XREAD` command syntax:
      ```
      XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]
      ```
      
      Here’s the structure for `XREAD`:
      ```json
      "arguments": [
                  {
                      "token": "COUNT",
                      "name": "count",
                      "type": "integer",
                      "optional": true
                  },
                  {
                      "token": "BLOCK",
                      "name": "milliseconds",
                      "type": "integer",
                      "optional": true
                  },
                  {
                      "name": "streams",
                      "token": "STREAMS",
                      "type": "block",
                      "arguments": [
                          {
                              "name": "key",
                              "type": "key",
                              "key_spec_index": 0,
                              "multiple": true
                          },
                          {
                              "name": "ID",
                              "type": "string",
                              "multiple": true
                          }
                      ]
                  }
      ]
      ```
      
      Now, consider the `HEXPIRE` syntax:
      ```
      HEXPIRE key seconds [NX | XX | GT | LT] FIELDS numfields field [field ...]
      ```
      
      Since the `FIELDS` token functions similarly to `STREAMS`, and given that `STREAMS` is defined as a block, I believe the `FIELDS` in `hepxire` should also be defined as a block.
      871c9859
  2. 27 May, 2024 1 commit
  3. 16 May, 2024 1 commit
  4. 03 May, 2024 1 commit
  5. 18 Apr, 2024 1 commit
    • Moti Cohen's avatar
      Hash Field Expiration - Basic support · c18ff056
      Moti Cohen authored
      - Add ebuckets & mstr data structures
      - Integrate active & lazy expiration
      - Add most of the commands 
      - Add support for dict (listpack is missing)
      TODOs:  RDB, notification, listpack, HSET, HGETF, defrag, aof
      c18ff056
  6. 29 Mar, 2023 1 commit
    • Itamar Haber's avatar
      Overhauls command summaries and man pages. (#11942) · 0c3b8b7e
      Itamar Haber authored
      This is an attempt to normalize/formalize command summaries.
      
      Main actions performed:
      
      * Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands.
      * Starts with "An internal command...", "A container command...", etc... when applicable.
      * Always uses periods.
      * Refrains from referring to other commands. If this is needed, backquotes should be used for command names.
      * Tries to be very clear about the data type when applicable.
      * Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed."
      * Prefers being terse over verbose.
      * Tries to be consistent.
      0c3b8b7e
  7. 11 Mar, 2023 1 commit
    • guybe7's avatar
      Add reply_schema to command json files (internal for now) (#10273) · 4ba47d2d
      guybe7 authored
      Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
      Since ironing the details of the reply schema of each and every command can take a long time, we
      would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
      Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
      
      ### Background
      In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
      
      ### Motivation
      1. Documentation. This is the primary goal.
      2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
        languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
      3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
        testsuite, see the "Testing" section)
      
      ### Schema
      The idea is to supply some sort of schema for the various replies of each command.
      The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
      Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
      and without the `FULL` modifier)
      We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
      
      Example for `BZPOPMIN`:
      ```
      "reply_schema": {
          "oneOf": [
              {
                  "description": "Timeout reached and no elements were popped.",
                  "type": "null"
              },
              {
                  "description": "The keyname, popped member, and its score.",
                  "type": "array",
                  "minItems": 3,
                  "maxItems": 3,
                  "items": [
                      {
                          "description": "Keyname",
                          "type": "string"
                      },
                      {
                          "description": "Member",
                          "type": "string"
                      },
                      {
                          "description": "Score",
                          "type": "number"
                      }
                  ]
              }
          ]
      }
      ```
      
      #### Notes
      1.  It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
        to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
        where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
      2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
        schema for documentation (and possibly to create a fuzzer that validates the replies)
      3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
        including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
        is with `WITHSCORES` and the other is without.
      4. For documentation, there will be another optional field "notes" in which we will add a short description of
        the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
        array, for example)
      
      Given the above:
      1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
        (given that "description" and "notes" are comprehensive enough)
      2. We can generate a client in a strongly typed language (but the return type could be a conceptual
        `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
      3. We can create a fuzzer for RESP3.
      
      ### Limitations (because we are using the standard json-schema)
      The problem is that Redis' replies are more diverse than what the json format allows. This means that,
      when we convert the reply to a json (in order to validate the schema against it), we lose information (see
      the "Testing" section below).
      The other option would have been to extend the standard json-schema (and json format) to include stuff
      like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
      seemed like too much work, so we decided to compromise.
      
      Examples:
      1. We cannot tell the difference between an "array" and a "set"
      2. We cannot tell the difference between simple-string and bulk-string
      3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
        case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
        compares (member,score) tuples and not just the member name. 
      
      ### Testing
      This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
      are indeed correct (i.e. describe the actual response of Redis).
      To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
      it executed and their replies.
      For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
      `--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
      `--log-req-res --force-resp3`)
      You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
      `.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
      These files are later on processed by `./utils/req-res-log-validator.py` which does:
      1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
      2. For each request-response pair, it validates the response against the request's reply_schema
        (obtained from the extended COMMAND DOCS)
      5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
        the existing redis test suite, rather than attempt to write a fuzzer.
      
      #### Notes about RESP2
      1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
        accept RESP3 as the future RESP)
      2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
        so that we can validate it, we will need to know how to convert the actual reply to the one expected.
         - number and boolean are always strings in RESP2 so the conversion is easy
         - objects (maps) are always a flat array in RESP2
         - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
           handling (so the client will not be totally auto-generated)
      
      Example for ZRANGE:
      ```
      "reply_schema": {
          "anyOf": [
              {
                  "description": "A list of member elements",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "string"
                  }
              },
              {
                  "description": "Members and their scores. Returned in case `WITHSCORES` was used.",
                  "notes": "In RESP2 this is returned as a flat array",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "array",
                      "minItems": 2,
                      "maxItems": 2,
                      "items": [
                          {
                              "description": "Member",
                              "type": "string"
                          },
                          {
                              "description": "Score",
                              "type": "number"
                          }
                      ]
                  }
              }
          ]
      }
      ```
      
      ### Other changes
      1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
        regardless of the special log-req-res mode ("Pub/Sub PING" for example)
      2. Update the history field of CLIENT LIST
      3. Added basic tests for commands that were not covered at all by the testsuite
      
      ### TODO
      
      - [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
        when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
        is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
      - [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
      - [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
      - [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
        of the tests - https://github.com/redis/redis/issues/11897
      - [x] (probably a separate PR) add all missing schemas
      - [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
      - [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
        fight with the tcl including mechanism a bit)
      - [x] issue: module API - https://github.com/redis/redis/issues/11898
      - [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
      
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarHanna Fadida <hanna.fadida@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarShaya Potter <shaya@redislabs.com>
      4ba47d2d
  8. 29 Jan, 2022 1 commit
    • Binbin's avatar
      commands arguments improvement about unix-time type (#10203) · 93d95155
      Binbin authored
      Change the name to unix-time-seconds or unix-time-milliseconds
      to be consistent. Change the type from INTEGER to UNIX_TIME.
      
      SET (EXAT and PXAT) was already ok.
      and naming aside, both PXAT and EXAT everywhere used unit-time (for both milliseconds and seconds).
      the only ones that where wrong are GETEX and XCLAIM (using "integer" for both seconds and milliseconds)
      93d95155
  9. 18 Jan, 2022 1 commit
    • Oran Agra's avatar
      New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122) · eef9c6b0
      Oran Agra authored
      The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
      explicit flags rather than just READ and WRITE. See discussion in #10040
      
      This PR defines two groups of flags:
      One about how redis internally handles the key (mutually-exclusive).
      The other is about the logical operation done from the user's point of view (3 mutually exclusive
      write flags, and one read flag, all optional).
      In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
      insert-only, we flag it as `RW` or `UPDATE`.
      here's the definition from the code:
      ```
      /* Key-spec flags *
       * -------------- */
      /* The following refer what the command actually does with the value or metadata
       * of the key, and not necessarily the user data or how it affects it.
       * Each key-spec may must have exaclty one of these. Any operation that's not
       * distinctly deletion, overwrite or read-only would be marked as RW. */
      #define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                        * doesn't necessarily returns it. */
      #define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                        * value of the key or its metadata. */
      #define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                        * the value of the key. */
      #define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
      /* The follwing refer to user data inside the value of the key, not the metadata
       * like LRU, type, cardinality. It refers to the logical operation on the user's
       * data (actual input strings / TTL), being used / returned / copied / changed,
       * It doesn't refer to modification or returning of metadata (like type, count,
       * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
       * Each key-spec may have one of the writes with or without access, or none: */
      #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                        * the value of the key. */
      #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                        * depend on the old value. */
      #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                        * modification or deletion of existing data. */
      #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                        * from the value of the key. */
      ```
      
      Unrelated changes:
      - generate-command-code.py is only compatible with python3 (modified the shabang)
      - generate-command-code.py print file on json parsing error
      - rename `shard_channel` key-spec flag to just `channel`.
      - add INCOMPLETE flag in input spec of SORT and SORT_RO
      eef9c6b0
  10. 29 Dec, 2021 1 commit
    • Itamar Haber's avatar
      Add missing metadata to the commands SSOT files. (#10016) · aec8c577
      Itamar Haber authored
      Add missing information about commands, mainly from reviewing redis-doc and removing
      the metadata from it (https://github.com/redis/redis-doc/pull/1722)
      
      * Reintroduces CLUSTER S****S (supported by Redis) but missing from the JSON / docs (related? #9675).
        Note that without that json file, the command won't work (breaking change)
      * Adds the `replicas` argument (exists in Redis) to `CLIENT KILL`.
      * Adds `history` entries to several commands based on redis-doc's man pages.
      * Adds `since` to applicable command arguments based on `history` (this basically makes
        some of `history` redundant - perhaps at a later stage).
      * Uses proper semantic versioning in all version references.
      * Also removes `geoencodeCommand` and `geodecodeCommand` header
        declarations per b96af595.
      aec8c577
  11. 15 Dec, 2021 1 commit
    • guybe7's avatar
      Auto-generate the command table from JSON files (#9656) · 86781600
      guybe7 authored
      Delete the hardcoded command table and replace it with an auto-generated table, based
      on a JSON file that describes the commands (each command must have a JSON file).
      
      These JSON files are the SSOT of everything there is to know about Redis commands,
      and it is reflected fully in COMMAND INFO.
      
      These JSON files are used to generate commands.c (using a python script), which is then
      committed to the repo and compiled.
      
      The purpose is:
      * Clients and proxies will be able to get much more info from redis, instead of relying on hard coded logic.
      * drop the dependency between Redis-user and the commands.json in redis-doc.
      * delete help.h and have redis-cli learn everything it needs to know just by issuing COMMAND (will be
        done in a separate PR)
      * redis.io should stop using commands.json and learn everything from Redis (ultimately one of the release
        artifacts should be a large JSON, containing all the information about all of the commands, which will be
        generated from COMMAND's reply)
      * the byproduct of this is:
        * module commands will be able to provide that info and possibly be more of a first-class citizens
        * in theory, one may be able to generate a redis client library for a strictly typed language, by using this info.
      
      ### Interface changes
      
      #### COMMAND INFO's reply change (and arg-less COMMAND)
      
      Before this commit the reply at index 7 contained the key-specs list
      and reply at index 8 contained the sub-commands list (Both unreleased).
      Now, reply at index 7 is a map of:
      - summary - short command description
      - since - debut version
      - group - command group
      - complexity - complexity string
      - doc-flags - flags used for documentation (e.g. "deprecated")
      - deprecated-since - if deprecated, from which version?
      - replaced-by - if deprecated, which command replaced it?
      - history - a list of (version, what-changed) tuples
      - hints - a list of strings, meant to provide hints for clients/proxies. see https://github.com/redis/redis/issues/9876
      - arguments - an array of arguments. each element is a map, with the possibility of nesting (sub-arguments)
      - key-specs - an array of keys specs (already in unstable, just changed location)
      - subcommands - a list of sub-commands (already in unstable, just changed location)
      - reply-schema - will be added in the future (see https://github.com/redis/redis/issues/9845)
      
      more details on these can be found in https://github.com/redis/redis-doc/pull/1697
      
      only the first three fields are mandatory 
      
      #### API changes (unreleased API obviously)
      
      now they take RedisModuleCommand opaque pointer instead of looking up the command by name
      
      - RM_CreateSubcommand
      - RM_AddCommandKeySpec
      - RM_SetCommandKeySpecBeginSearchIndex
      - RM_SetCommandKeySpecBeginSearchKeyword
      - RM_SetCommandKeySpecFindKeysRange
      - RM_SetCommandKeySpecFindKeysKeynum
      
      Currently, we did not add module API to provide additional information about their commands because
      we couldn't agree on how the API should look like, see https://github.com/redis/redis/issues/9944
      
      .
      
      ### Somehow related changes
      1. Literals should be in uppercase while placeholder in lowercase. Now all the GEO* command
         will be documented with M|KM|FT|MI and can take both lowercase and uppercase
      
      ### Unrelated changes
      1. Bugfix: no_madaory_keys was absent in COMMAND's reply
      2. expose CMD_MODULE as "module" via COMMAND
      3. have a dedicated uint64 for ACL categories (instead of having them in the same uint64 as command flags)
      Co-authored-by: default avatarItamar Haber <itamar@garantiadata.com>
      86781600