- 11 Mar, 2023 1 commit
-
-
guybe7 authored
Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845 Since ironing the details of the reply schema of each and every command can take a long time, we would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch. Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build. ### Background In #9656 we add a lot of information about Redis commands, but we are missing information about the replies ### Motivation 1. Documentation. This is the primary goal. 2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like. 3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing testsuite, see the "Testing" section) ### Schema The idea is to supply some sort of schema for the various replies of each command. The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3. Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with and without the `FULL` modifier) We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema. Example for `BZPOPMIN`: ``` "reply_schema": { "oneOf": [ { "description": "Timeout reached and no elements were popped.", "type": "null" }, { "description": "The keyname, popped member, and its score.", "type": "array", "minItems": 3, "maxItems": 3, "items": [ { "description": "Keyname", "type": "string" }, { "description": "Member", "type": "string" }, { "description": "Score", "type": "number" } ] } ] } ``` #### Notes 1. It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI, where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one. 2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply schema for documentation (and possibly to create a fuzzer that validates the replies) 3. For documentation, the description field will include an explanation of the scenario in which the reply is sent, including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one is with `WITHSCORES` and the other is without. 4. For documentation, there will be another optional field "notes" in which we will add a short description of the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat array, for example) Given the above: 1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/) (given that "description" and "notes" are comprehensive enough) 2. We can generate a client in a strongly typed language (but the return type could be a conceptual `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support. 3. We can create a fuzzer for RESP3. ### Limitations (because we are using the standard json-schema) The problem is that Redis' replies are more diverse than what the json format allows. This means that, when we convert the reply to a json (in order to validate the schema against it), we lose information (see the "Testing" section below). The other option would have been to extend the standard json-schema (and json format) to include stuff like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that seemed like too much work, so we decided to compromise. Examples: 1. We cannot tell the difference between an "array" and a "set" 2. We cannot tell the difference between simple-string and bulk-string 3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems` compares (member,score) tuples and not just the member name. ### Testing This commit includes some changes inside Redis in order to verify the schemas (existing and future ones) are indeed correct (i.e. describe the actual response of Redis). To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands it executed and their replies. For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with `--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with `--log-req-res --force-resp3`) You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate `.reqres` files (same dir as the `stdout` files) which contain request-response pairs. These files are later on processed by `./utils/req-res-log-validator.py` which does: 1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c) 2. For each request-response pair, it validates the response against the request's reply_schema (obtained from the extended COMMAND DOCS) 5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use the existing redis test suite, rather than attempt to write a fuzzer. #### Notes about RESP2 1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to accept RESP3 as the future RESP) 2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3 so that we can validate it, we will need to know how to convert the actual reply to the one expected. - number and boolean are always strings in RESP2 so the conversion is easy - objects (maps) are always a flat array in RESP2 - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command handling (so the client will not be totally auto-generated) Example for ZRANGE: ``` "reply_schema": { "anyOf": [ { "description": "A list of member elements", "type": "array", "uniqueItems": true, "items": { "type": "string" } }, { "description": "Members and their scores. Returned in case `WITHSCORES` was used.", "notes": "In RESP2 this is returned as a flat array", "type": "array", "uniqueItems": true, "items": { "type": "array", "minItems": 2, "maxItems": 2, "items": [ { "description": "Member", "type": "string" }, { "description": "Score", "type": "number" } ] } } ] } ``` ### Other changes 1. Some tests that behave differently depending on the RESP are now being tested for both RESP, regardless of the special log-req-res mode ("Pub/Sub PING" for example) 2. Update the history field of CLIENT LIST 3. Added basic tests for commands that were not covered at all by the testsuite ### TODO - [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g. when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896 - [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode - [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator) - [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output of the tests - https://github.com/redis/redis/issues/11897 - [x] (probably a separate PR) add all missing schemas - [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res - [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to fight with the tcl including mechanism a bit) - [x] issue: module API - https://github.com/redis/redis/issues/11898 - [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899 Co-authored-by:
Ozan Tezcan <ozantezcan@gmail.com> Co-authored-by:
Hanna Fadida <hanna.fadida@redislabs.com> Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Shaya Potter <shaya@redislabs.com>
-
- 23 Feb, 2023 2 commits
-
-
Binbin authored
CLIENT NO-TOUCH added in #11483, but forgot to add the since field in the JSON file. This PR adds the since field to it with a value of 7.2.0
-
Chen Tianjie authored
When no-touch mode is enabled, the client will not touch LRU/LFU of the keys it accesses, except when executing command `TOUCH`. This allows inspecting or modifying the key-space without affecting their eviction. Changes: - A command `CLIENT NO-TOUCH ON|OFF` to switch on and off this mode. - A client flag `#define CLIENT_NOTOUCH (1ULL<<45)`, which can be shown with `CLIENT INFO`, by the letter "T" in the "flags" field. - Clear `NO-TOUCH` flag in `clearClientConnectionState`, which is used by `RESET` command and resetting temp clients used by modules. - Also clear `NO-EVICT` flag in `clearClientConnectionState`, this might have been an oversight, spotted by @madolson. - A test using `DEBUG OBJECT` command to verify that LRU stat is not touched when no-touch mode is on. Co-authored-by:
chentianjie <chentianjie@alibaba-inc.com> Co-authored-by:
Madelyn Olson <34459052+madolson@users.noreply.github.com> Co-authored-by:
sundb <sundbcn@gmail.com>
-
- 14 Feb, 2023 1 commit
-
-
guybe7 authored
Starting from Redis 7.0 (#9890) we started wrapping everything a command propagates with MULTI/EXEC. The problem is that both SCAN and RANDOMKEY can lazy-expire arbitrary keys (similar behavior to active-expire), and put DELs in a transaction. Fix: When these commands are called without a parent exec-unit (e.g. not in EVAL or MULTI) we avoid wrapping their DELs in a transaction (for the same reasons active-expire and eviction avoids a transaction) This PR adds a per-command flag that indicates that the command may touch arbitrary keys (not the ones in the arguments), and uses that flag to avoid the MULTI-EXEC. For now, this flag is internal, since we're considering other solutions for the future. Note for cluster mode: if SCAN/RANDOMKEY is inside EVAL/MULTI it can still cause the same situation (as it always did), but it won't cause a CROSSSLOT because replicas and AOF do not perform slot checks. The problem with the above is mainly for 3rd party ecosystem tools that propagate commands from master to master, or feed an AOF file with redis-cli into a master. This PR aims to fix the regression in redis 7.0, and we opened #11792 to try to handle the bigger problem with lazy expire better for another release.
-
- 02 Feb, 2023 1 commit
-
-
Roshan Khatri authored
Added 3 fields to the ACL LOG - adds entry_id, timestamp_created and timestamp_last_updated, which updates similar existing log error entries. The pair - entry_id, timestamp_created is a unique identifier of this entry, in case the node dies and is restarted, it can detect that if it's a new series. The primary use case of Unique id is to uniquely identify the error messages and not to detect if the server has restarted. entry-id is the sequence number of the entry (starting at 0) since the server process started. Can also be used to check if items were "lost" if they fell between periods. timestamp-created is the unix-time in ms at the time the entry was first created. timestamp-last-updated is the unix-time in ms at the time the entry was last updated Time_created gives the absolute time which better accounts for network time as compared to time since. It can also be older than 60 secs and presently there is no field that can display the original time of creation once the error entry is updated. The reason of timestamp_last_updated field is that it provides a more precise value for the “last time” an error was seen where as, presently it is only in the 60 second period. Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 01 Feb, 2023 1 commit
-
-
Binbin authored
Change history: - `user` added in 6.0.0, 0f42447a - `argv-mem` and `tot-mem` added in 6.2.0, bea40e6a - `redir` added in 6.2.0, dd1f20ed - `resp` added in 7.0.0, 7c376398 - `multi-mem` added in 7.0.0, 2753429c - `rbs` and `rbp` added in 7.0.0, 47c51d0c - `ssub` added in 7.0.3, 35c2ee87
-
- 26 Jan, 2023 1 commit
-
-
Wen Hui authored
The command: sentinel config set option value and sentinel config get option They should include at least 4 arguments instead of 3, This PR fixes this issue. the only impact on the client is a different error message
-
- 25 Jan, 2023 2 commits
- 10 Jan, 2023 1 commit
-
-
knggk authored
the metadata for the new arguments of XSETID, entries-added and max-deleted-id, which have been added in Redis 7.0 was missing.
-
- 28 Nov, 2022 1 commit
-
-
C Charles authored
Add an option "withscores" to ZRANK and ZREVRANK. Add `[withscore]` option to both `zrank` and `zrevrank`, like this: ``` z[rev]rank key member [withscore] ```
-
- 24 Nov, 2022 1 commit
-
-
Wen Hui authored
Command SENTINEL DEBUG could be no arguments, which display all configurable arguments and their values. Update the command arguments in the docs (json file) to indicate that arguments are optional
-
- 22 Nov, 2022 1 commit
-
-
Itamar Haber authored
Technically, these commands were deprecated as of 2.6.12, with the introduction of the respective arguments to SET. In reality, the deprecation note will only be added in 7.2.0.
-
- 17 Nov, 2022 1 commit
-
-
Ping Xie authored
Introduce Shard IDs to logically group nodes in cluster mode. 1. Added a new "shard_id" field to "cluster nodes" output and nodes.conf after "hostname" 2. Added a new PING extension to propagate "shard_id" 3. Handled upgrade from pre-7.2 releases automatically 4. Refactored PING extension assembling/parsing logic Behavior of Shard IDs: Replicas will always follow the shards of their reported primaries. If a primary updates its shard ID, the replica will follow. (This need not follow for cluster v2) This is not an expected use case.
-
- 09 Nov, 2022 1 commit
-
-
Viktor Söderqvist authored
Clients should not use this command. Instead, clients should simply close the connection when they're not used anymore. Terminating a connection on the client side is preferable, as it eliminates `TIME_WAIT` lingering sockets on the server side.
-
- 02 Nov, 2022 1 commit
-
-
Wen Hui authored
According to the source code, the commands can be executed with only key name, and no GET/SET/INCR operation arguments. change the docs to reflect that by marking these arguments as optional. also add tests.
-
- 25 Oct, 2022 1 commit
-
-
Wen Hui authored
These commands take a list of members, which can be empty (i.e. running the command with just a key name). this always results in an empty array reply, so it doesn't make much sense, but changing it is a breaking change. This PR fixes the documentation, making the member field as optional, just makes sure the command format documentation is consistent with the command behavior. The command format will be: 127.0.0.1:6381> GEOPOS key [member [member ...]] 127.0.0.1:6381> GEOHASH key [member [member ...]]
-
- 22 Oct, 2022 1 commit
-
-
Binbin authored
Make PFMERGE source key optional in docs, add tests with one input key, add tests on missing source keys (#11205) The following two cases will create an empty destkey HLL: 1. called with no source keys, like `pfmerge destkey` 2. called with non-existing source keys, like `pfmerge destkey non-existing-source-key` In the first case, in `PFMERGE`, the dest key is actually one of the source keys too. So `PFMERGE k1 k2` is equivalent to `SUNIONSTORE k1 k1 k2`, and `PFMERGE k1` is equivalent to `SUNIONSTORE k1 k1`. So the first case is reasonable, the source key is actually optional. And the second case, `PFMERGE` on missing keys should succeed and create an empty dest. This is consistent with `PFCOUNT`, and also with `SUNIONSTORE`, no need to change.
-
- 11 Oct, 2022 1 commit
-
-
Nikita Tolkachev authored
remove double period at the end of sentence.
-
- 02 Oct, 2022 2 commits
-
-
Binbin authored
- fix `the the` typo - `LPOPRPUSH` does not exist, should be `RPOPLPUSH` - `CLUSTER GETKEYINSLOT` 's time complexity should be O(N) - `there bytes` should be `three bytes`, this closes #11266 - `slave` word to `replica` in log, modified the front and missed the back - remove useless aofReadDiffFromParent in server.h - `trackingHandlePendingKeyInvalidations` method adds a void parameter
-
Huang Zhw authored
- BITOP: turn argument `operation` from string to oneof - CLIENT KILL: turn argument `skipme` from string to oneof - COMMAND GETKEYS / GETKEYSANDFLAGS: change arguments to optional, and change arity to -3 (fixes regression in redis 7.0) - CLIENT PAUSE: this command was added in v3.0.0
-
- 29 Sep, 2022 1 commit
-
-
Wen Hui authored
The sentinel CONFIG GET command doesn't support multiple arguments, but the json file did. remove that.
-
- 08 Sep, 2022 1 commit
-
-
Wen Hui authored
For the stream data type, some commands, such as **XGROUP CREATE, XGROUP DESTROY, XGROUP CREATECONSUMER, XGROUP DELCONSUMER and XINFO CONSUMERS** use groupname and consumername in the command description; However, for the commands **XREADGROUP GROUP, XPENDING, XACK , XCLAIM and XAUTOCLAIM** use term "group and consumer", clients could be confused. This PR goal is to unify all the commands to groupname and consumername.
-
- 28 Aug, 2022 1 commit
-
-
Huang Zhw authored
TLDR: the CLUSTER command originally had the `random` flag, so all the sub-commands initially got that new flag, but in fact many of them don't need it. The only effect of this change is on the output of COMMAND INFO.
-
- 22 Aug, 2022 1 commit
-
-
Itamar Haber authored
-
- 21 Aug, 2022 2 commits
-
-
Itamar Haber authored
A overlooked mistake in the #11162
-
Itamar Haber authored
This change was part of #9656 (Redis 7.0)
-
- 18 Aug, 2022 1 commit
-
-
guybe7 authored
This PR makes sure that "name" is unique for all arguments in the same level (i.e. all args of a command and all args within a block/oneof). This means several argument with identical meaning can be referred to together, but also if someone needs to refer to a specific one, they can use its full path. In addition, the "display_text" field has been added, to be used by redis.io in order to render the syntax of the command (for the vast majority it is identical to "name" but sometimes we want to use a different string that is not "name") The "display" field is exposed via COMMAND DOCS and will be present for every argument, except "oneof" and "block" (which are container arguments) Other changes: 1. Make sure we do not have any container arguments ("oneof" or "block") that contain less than two sub-args (otherwise it doesn't make sense) 2. migrate.json: both AUTH and AUTH2 should not be "optional" 3. arg names cannot contain underscores, and force the usage of hyphens (most of these were a result of the script that generated the initial json files from redis.io commands.json).
-
- 01 Aug, 2022 1 commit
-
-
Rudi Floren authored
The docs state that there is a new and an old argument format. The current state of the arguments allows mixing the old and new format, thus the need for two additional oneof blocks. One for differentiating the new from the old format and then one to allow setting multiple filters using the new format.
-
- 24 Jul, 2022 1 commit
-
-
Pavel Krush authored
Swap M and N in the complexity formula of [B]ZMPOP Co-authored-by:
Pavel Krush <neon@pushwoosh.com>
-
- 20 Jul, 2022 1 commit
-
-
guybe7 authored
It seems it was overlooked when we first created the json files
-
- 19 Jul, 2022 1 commit
-
-
Wen Hui authored
these are missing from the RO_ commands, present in the other ones. Co-authored-by:
Ubuntu <lucas.guang.yang1@huawei.com>
-
- 17 Jul, 2022 1 commit
-
-
Wen Hui authored
According to the Redis functions documentation, FCALL command format could be FCALL function_name numberOfKeys [key1, key2, key3.....] [arg1, arg2, arg3.....] So in the json file of fcall and fcall_ro, we should add optional for key and arg part. Just like EVAL... Co-authored-by:
Binbin <binloveplay1314@qq.com>
-
- 30 Jun, 2022 1 commit
-
-
Binbin authored
This was harmless because we marked the parent command with SENTINEL flag. So the populateCommandTable was ok. And we also don't show the flag (SENTINEL and ONLY-SENTNEL) in COMMAND INFO. In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL flags check when populating the sub-commands. so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
-
- 29 Jun, 2022 1 commit
-
-
Wen Hui authored
Fix regression of CLUSTER RESET command in redis 7.0. cluster reset command format is: CLUSTER RESET [ HARD | SOFT] According to the cluster reset command doc and codes, the third argument is optional, so the arity in json file should be -2 instead of 3. Add test to verify future regressions with RESET and RESET SOFT that were not covered. Co-authored-by:
Ubuntu <lucas.guang.yang1@huawei.com> Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Binbin <binloveplay1314@qq.com>
-
- 25 Jun, 2022 1 commit
-
-
Steve Lorello authored
In 6.2.0 with the introduction of the REV subcommand in ZRANGE, there was a semantic shift in the arguments of ZRANGE when the REV sub-command is executed. Without the sub-command `min` and `max` (the old names of the arguments) are appropriate because if you put the min value and the max value in everything works fine. ```bash 127.0.0.1:6379> ZADD myset 0 foo (integer) 1 127.0.0.1:6379> ZADD myset 1 bar (integer) 1 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE 1) "foo" 2) "bar" ``` However - if you add the `REV` subcommand, ordering the arguments `min` `max` breaks the command: ```bash 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE REV (empty array) ``` why? because `ZRANGE` with the `REV` sub-command is expecting the `max` first and the `min` second (because it's a reverse range like `ZREVRANGEBYSCORE`): ```bash 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE REV (empty array) ```
-
- 23 Jun, 2022 1 commit
-
- 14 Jun, 2022 1 commit
-
-
Oran Agra authored
I noticed that scripting.tcl uses INFO from within a script and thought it's an overkill and concluded it's nicer to use another CMD_STALE command, decided to use ECHO, and then noticed it's not at all allowed in stale mode. probably overlooked at #6843
-
- 07 Jun, 2022 2 commits
-
-
Bjorn Svensson authored
Correcting the introduction version of the command `BITFIELD_RO` Command added by commit: b3e4abf0 Add history info of the `FULL` modifier in `XINFO STREAM` Modifier was added by commit: 1e2aee39 (Includes output from `./utils/generate-command-code.py`)
-
Binbin authored
Currently generate-command.help.rb dose not handle the multiple_token flag, handle this flag in this PR. The format is the same as redis-cli rendering. ```diff - bitfield_ro key GET encoding offset [encoding offset ...] + bitfield_ro key GET encoding offset [GET encoding offset ...] ``` Re run generate-command-code.py which was forget in #10820. Also change the flag value from string to bool, like "true" to true
-