- 08 May, 2024 1 commit
-
-
Ozan Tezcan authored
**Changes:** - Adds listpack support to hash field expiration - Implements hgetf/hsetf commands **Listpack support for hash field expiration** We keep field name and value pairs in listpack for the hash type. With this PR, if one of hash field expiration command is called on the key for the first time, it converts listpack layout to triplets to hold field name, value and ttl per field. If a field does not have a TTL, we store zero as the ttl value. Zero is encoded as two bytes in the listpack. So, once we convert listpack to hold triplets, for the fields that don't have a TTL, it will be consuming those extra 2 bytes per item. Fields are ordered by ttl in the listpack to find the field with minimum expiry time efficiently. **New command implementations as part of this PR:** - HGETF command For each specified field get its value and optionally set the field's expiration time in sec/msec /unix-sec/unix-msec: ``` HGETF key [NX | XX | GT | LT] [EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds | PERSIST] <FIELDS count field [field ...]> ``` - HSETF command For each specified field value pair: set field to value and optionally set the field's expiration time in sec/msec /unix-sec/unix-msec: ``` HSETF key [DC] [DCF | DOF] [NX | XX | GT | LT] [GETNEW | GETOLD] [EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT unix-time-milliseconds | KEEPTTL] <FVS count field value [field value …]> ``` Todo: - Performance improvement. - rdb load/save - aof - defrag
-
- 03 May, 2024 1 commit
-
-
debing.sun authored
-
- 18 Apr, 2024 1 commit
-
-
Moti Cohen authored
- Add ebuckets & mstr data structures - Integrate active & lazy expiration - Add most of the commands - Add support for dict (listpack is missing) TODOs: RDB, notification, listpack, HSET, HGETF, defrag, aof
-
- 01 Mar, 2024 1 commit
-
-
Chen Tianjie authored
Sometimes we need to make fast judgement about why Redis is suddenly taking more memory. One of the reasons is main DB's dicts doing rehashing. We may use `MEMORY STATS` to monitor the overhead memory of each DB, but there still lacks a total sum to show an overall trend. So this PR adds the total overhead of all DBs to `INFO MEMORY` section, together with the total count of rehashing DB dicts, providing some intuitive metrics about main dicts rehashing. This PR adds the following metrics to INFO MEMORY * `mem_overhead_db_hashtable_rehashing` - only size of ht[0] in dictionaries we're rehashing (i.e. the memory that's gonna get released soon) and a similar ones to MEMORY STATS: * `overhead.db.hashtable.lut` (complements the existing `overhead.hashtable.main` and `overhead.hashtable.expires` which also counts the `dictEntry` structs too) * `overhead.db.hashtable.rehashing` - temporary rehashing overhead. * `db.dict.rehashing.count` - number of top level dictionaries being rehashed. --------- Co-authored-by:
zhaozhao.zz <zhaozhao.zz@alibaba-inc.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 29 Feb, 2024 1 commit
-
-
Binbin authored
In XREADGROUP ACK, because streamPropagateXCLAIM does not propagate entries-read, entries-read will be inconsistent between master and replicas. I.e. if no entries were claimed, it would have propagated correctly, but if some were claimed, then the entries-read field would be inconsistent on the replica. The fix was suggested by guybe7, call streamPropagateGroupID unconditionally, so that we will normalize entries_read on the replicas. In the past, we would only set propagate_last_id when NOACK was specified. And in #9127, XCLAIM did not propagate entries_read in ACK, which would cause entries_read to be inconsistent between master and replicas. Another approach is add another arg to XCLAIM and let it propagate entries_read, but we decided not to use it. Because we want minimal damage in case there's an old target and new source (in the worst case scenario, the new source doesn't recognize XGROUP SETID ... ENTRIES READ and the lag is lost. If we change XCLAIM, the damage is much more severe). In this patch, now if the user uses XREADGROUP .. COUNT 1 there will be an additional overhead of MULTI, EXEC and XGROUPSETID. We assume the extra command in case of COUNT 1 (4x factor, changing from one XCLAIM to MULTI+XCLAIM+XSETID+EXEC), is probably ok since reading just one entry is in any case very inefficient (a client round trip per record), so we're hoping it's not a common case. Issue was introduced in #9127.
-
- 22 Feb, 2024 1 commit
-
-
guybe7 authored
Now it matches the information in xinfo-stream.json
-
- 21 Feb, 2024 1 commit
-
-
Binbin authored
This field was added in #12996 but forgot to add it in json file. This also causes reply-schemas-validator to fail.
-
- 20 Feb, 2024 1 commit
-
-
Binbin authored
Recently I saw in CI that reply-schemas-validator fails here: ``` Failed validating 'minimum' in schema[1]['properties']['groups']['items']['properties']['consumers']['items']['properties']['active-time']: {'description': 'Last time this consumer was active (successful ' 'reading/claiming).', 'minimum': 0, 'type': 'integer'} On instance['groups'][0]['consumers'][0]['active-time']: -1729380548878722639 ``` The reason is that in fuzzer, we may restore corrupted active-time, which will cause the reply schema CI to fail. The fuzzer can cause corrupt the state in many places, which will bugs that mess up the reply, so we decided to skip logreqres. Also, seen-time is the same type as active-time, adding the minimum. --------- Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 19 Feb, 2024 1 commit
-
-
guybe7 authored
Add readme about the command json folder, what it does, and who should (not) use it. see discussion https://github.com/redis/redis/issues/9359#issuecomment-1936420698 --------- Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Binbin <binloveplay1314@qq.com>
-
- 04 Feb, 2024 1 commit
-
-
Daz authored
The JSON file lacks the following structural API changes: - GEORADIUSBYMEMBER: add the ANY option for COUNT since 6.2.0. - GEORADIUSBYMEMBER_RO: add the ANY option for COUNT since 6.2.0. - GEORADIUS_RO: Added support for uppercase unit names since 7.0.0. - GEORADIUSBYMEMBER_RO: Added support for uppercase unit names since 7.0.0. --------- Signed-off-by:
daz-3ux <daz-3ux@proton.me> Co-authored-by:
bodong.ybd <bodong.ybd@alibaba-inc.com> Co-authored-by:
Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by:
yangpengda.333 <yangpengda.333@bytedance.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 30 Jan, 2024 2 commits
-
-
Chen Tianjie authored
Add a way to HSCAN a hash key, and get only the filed names. Command syntax is now: ``` HSCAN key cursor [MATCH pattern] [COUNT count] [NOVALUES] ``` when `NOVALUES` is on, the command will only return keys in the hash. --------- Co-authored-by:
Viktor Söderqvist <viktor.soderqvist@est.tech>
-
Slava Koyfman authored
Adds an ability to kill clients older than a specified age. Also, fixed the age calculation in `catClientInfoString` to use `commandTimeSnapshot` instead of the old `server.unixtime`, and added missing documentation for `CLIENT KILL ID` to output of `CLIENT help`. --------- Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 23 Jan, 2024 1 commit
-
-
Binbin authored
In #11568 we removed the NOSCRIPT flag from commands, e.g. removing NOSCRIPT flag from WAIT. Aiming to allow them in scripts and let them implicitly behave in the non-blocking way. This PR remove NOSCRIPT flag from WAITAOF just like WAIT (to be symmetrical)). And this PR also add BLOCKING flag for WAIT and WAITAOF.
-
- 08 Jan, 2024 1 commit
-
-
debing.sun authored
In #10122, we set the destination key's flag of SINTERSTORE to `RW`, however, this command doesn't actually read or modify the destination key, just overwrites it. Therefore, we change it to `OW` similarly to all other *STORE commands.
-
- 10 Dec, 2023 1 commit
-
-
Binbin authored
overhead.hashtable.slot-to-keys was added in 7.0 in #10017, then removed in #11695. Now remove it from reply_schema.
-
- 12 Oct, 2023 1 commit
-
-
zhaozhao.zz authored
In #11568 we removed the NOSCRIPT flag from commands and keep the BLOCKING flag. Aiming to allow them in scripts and let them implicitly behave in the non-blocking way. In that sense, the old behavior was to allow LPOP and reject BLPOP, and the new behavior, is to allow BLPOP too, and fail it only in case it ends up blocking. So likewise, so far we allowed XREAD and rejected XREAD BLOCK, and we will now allow that too, and only reject it if it ends up blocking.
-
- 10 Oct, 2023 1 commit
-
-
Binbin authored
The current commands.json doesn't mention the special NO ONE arguments. This change is also applied to SLAVEOF
-
- 10 Sep, 2023 1 commit
-
-
Binbin authored
An unintentional change was introduced in #10536, we used to use addReplyLongLong and now it is addReplyBulkLonglong, revert it back the previous behavior.
-
- 04 Sep, 2023 1 commit
-
-
nihohit authored
Updated the command tips for ACL SAVE / SETUSER / DELUSER, CLIENT SETNAME / SETINFO, and LATENCY RESET. The tips now match CONFIG SET, since there's a similar behavior for all of these commands - the user expects to update the various configurations & states on all nodes, not only on a single, random node. For LATENCY RESET the response tip is now agg_sum. Co-authored-by:
Shachar Langbeheim <shachlan@amazon.com>
-
- 31 Aug, 2023 1 commit
-
-
Binbin authored
Also added a test to cover this case, so this can cover the reply schemas check.
-
- 30 Aug, 2023 1 commit
-
-
nihohit authored
Since the three commands have similar behavior (change config, return OK), the tips that govern how they should behave should be similar. Co-authored-by:
Shachar Langbeheim <shachlan@amazon.com>
-
- 15 Aug, 2023 1 commit
-
-
Binbin authored
We iterate over all replicas to get the result, the time complexity should be O(N), like CLUSTER NODES complexity is O(N).
-
- 05 Aug, 2023 1 commit
-
-
Binbin authored
GEOHASH / GEODIST / GEOPOS use zsetScore to get the score, in skiplist encoding, we use dictFind to get the score, which is O(1), same as ZSCORE command. It is not clear why these commands had O(Log(N)), and O(N) until now.
-
- 25 Jul, 2023 1 commit
-
-
nihohit authored
changing the response and request policy of a few commands, see https://redis.io/docs/reference/command-tips 1. RANDOMKEY used to have no response policy, which means that when sent to multiple shards, the responses should be aggregated. this normally applies to commands that return arrays, but since RANDOMKEY replies with a simple string, it actually requires a SPECIAL response policy (for the client to select just one) 2. SCAN used to have no response policy, but although the key names part of the response can be aggregated, the cursor part certainly can't. 3. MSETNX had a request policy of MULTI_SHARD and response policy of AGG_MIN, but in fact the contract with MSETNX is that when one key exists, it returns 0 and doesn't set any key, routing it to multiple shards would mean that if one failed and another succeeded, it's atomicity is broken and it's impossible to return a valid response to the caller. Co-authored-by:
Shachar Langbeheim <shachlan@amazon.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 20 Jun, 2023 1 commit
-
-
Binbin authored
The parameter name is WITHSCORE instead of WITHSCORES.
-
- 19 Jun, 2023 1 commit
-
-
Binbin authored
In the original implementation, the time complexity of the commands is actually O(N*M), where N is the number of patterns the client is already subscribed and M is the number of patterns to subscribe to. The docs are all wrong about this. Specifically, because the original client->pubsub_patterns is a list, so we need to do listSearchKey which is O(N). In this PR, we change it to a dict, so the search becomes O(1). At the same time, both pubsub_channels and pubsubshard_channels are dicts. Changing pubsub_patterns to a dictionary improves the readability and maintainability of the code.
-
- 13 Jun, 2023 1 commit
-
-
Harkrishn Patro authored
It would be helpful for clients to get cluster slots/shards information during a node failover and is loading data.
-
- 26 May, 2023 1 commit
-
-
Binbin authored
It was missing in #12223, and the reply-schemas daily was failing: ``` jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas Failed validating 'oneOf' in schema[0]['properties']['loglevel']: {'oneOf': [{'const': 'debug'}, {'const': 'verbose'}, {'const': 'notice'}, {'const': 'warning'}, {'const': 'unknown'}]} On instance['loglevel']: 'nothing' ```
-
- 17 May, 2023 1 commit
-
-
Wen Hui authored
Extend SENTINEL CONFIG SET and SENTINEL CONFIG GET to be compatible with variadic CONFIG SET and CONFIG GET and allow multiple parameters to be modified in a single call atomically. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 10 May, 2023 1 commit
-
-
Binbin authored
This pattern is from COMMAND INFO: Returns information about one, multiple or all commands. Also re-generate commands.def, the GEO change was missing in #12151.
-
- 09 May, 2023 1 commit
-
-
Leibale Eidelman authored
in GEO commands, `STORE` and `STOREDIST` are mutually exclusive. use `oneof` block to contain them
-
- 08 May, 2023 1 commit
-
-
cui fliter authored
Remove several instances of duplicate "the" in comments
-
- 27 Apr, 2023 1 commit
-
-
Binbin authored
The change in #12018 break the CI (fixed by #12083). There are quite a few sentinel commands that are missing both test coverage and also schema. PR added reply-schema to the following commands: - sentinel debug - sentinel info-cache - sentinel pendding-scripts - sentinel reset - sentinel simulate-failure Added some very basic tests for other sentinel commands, just so that they have some coverage. - sentinel help - sentinel masters - sentinel myid - sentinel sentinels - sentinel slaves These tests should be improved / replaced in a followup PR.
-
- 20 Apr, 2023 1 commit
-
-
Juho Kim authored
"lst" to "last" in commands help
-
- 19 Apr, 2023 1 commit
-
-
Wen Hui authored
Some sentinel subcommands are missing the reply_schema in the json file, so add the proper reply_schema part in json file as sentinel replicas commands. The schema validator was skipping coverage test for sentinel commands, this was initially done just in order to focus on redis commands and leave sentinel coverage for later, so this check is now removed. sentinel commands that were missing reply schema: * sentinel masters * sentinel myid * sentinel sentinels <master-name> * sentinel slaves (deprecated) <master-name>
-
- 13 Apr, 2023 1 commit
-
-
Wen Hui authored
These commands had an empty description. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 29 Mar, 2023 1 commit
-
-
Itamar Haber authored
This is an attempt to normalize/formalize command summaries. Main actions performed: * Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands. * Starts with "An internal command...", "A container command...", etc... when applicable. * Always uses periods. * Refrains from referring to other commands. If this is needed, backquotes should be used for command names. * Tries to be very clear about the data type when applicable. * Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed." * Prefers being terse over verbose. * Tries to be consistent.
-
- 22 Mar, 2023 2 commits
-
-
Oran Agra authored
-
Igor Malinovskiy authored
This PR allows clients to send information about the client library to redis to be displayed in CLIENT LIST and CLIENT INFO. Currently supports: `CLIENT [lib-name | lib-ver] <value>` Client libraries are expected to pipeline these right after AUTH, and ignore the failure in case they're talking to an older version of redis. These will be shown in CLIENT LIST and CLIENT INFO as: * `lib-name` - meant to hold the client library name. * `lib-ver` - meant to hold the client library version. The values cannot contain spaces, newlines and any wild ASCII characters, but all other normal chars are accepted, e.g `.`, `=` etc (same as CLIENT NAME). The RESET command does NOT clear these, but they can be cleared to the default by sending a command with a blank string. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 16 Mar, 2023 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Allow running blocking commands from within a module using `RM_Call`. Today, when `RM_Call` is used, the fake client that is used to run command is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command that it is not allowed to block the client and in case it needs to block, it must fallback to some alternative (either return error or perform some default behavior). For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block. All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including module commands). When the command invocation finished, Redis asserts that the client was not blocked. This PR introduces the ability to call blocking command using `RM_Call` by passing a callback that will be called when the client will get unblocked. In order to do that, the user must explicitly say that he allow to perform blocking command by passing a new format specifier argument, `K`, to the `RM_Call` function. This new flag will tell Redis that it is allow to run blocking command and block the client. In case the command got blocked, Redis will return a new type of call reply (`REDISMODULE_REPLY_PROMISE`). This call reply indicates that the command got blocked and the user can set the on_unblocked handler using `RM_CallReplyPromiseSetUnblockHandler`. When clients gets unblocked, it eventually reaches `processUnblockedClients` function. This is where we check if the client is a fake module client and if it is, we call the unblock callback instead of performing the usual unblock operations. **Notice**: `RM_CallReplyPromiseSetUnblockHandler` must be called atomically along side the command invocation (without releasing the Redis lock in between). In addition, unlike other CallReply types, the promise call reply must be released by the module when the Redis GIL is acquired. The module can abort the execution on the blocking command (if it was not yet executed) using `RM_CallReplyPromiseAbort`. the API will return `REDISMODULE_OK` on success and `REDISMODULE_ERR` if the operation is already executed. **Notice** that in case of misbehave module, Abort might finished successfully but the operation will not really be aborted. This can only happened if the module do not respect the disconnect callback of the blocked client. For pure Redis commands this can not happened. ### Atomicity Guarantees The API promise that the unblock handler will run atomically as an execution unit. This means that all the operation performed on the unblock handler will be wrapped with a multi exec transaction when replicated to the replica and AOF. The API **do not** grantee any other atomicity properties such as when the unblock handler will be called. This gives us the flexibility to strengthen the grantees (or not) in the future if we will decide that we need a better guarantees. That said, the implementation **does** provide a better guarantees when performing pure Redis blocking command like `BLPOP`. In this case the unblock handler will run atomically with the operation that got unblocked (for example, in case of `BLPOP`, the unblock handler will run atomically with the `LPOP` operation that run when the command got unblocked). This is an implementation detail that might be change in the future and the module writer should not count on that. ### Calling blocking commands while running on script mode (`S`) `RM_Call` script mode (`S`) was introduced on #0372. It is used for usecases where the command that was invoked on `RM_Call` comes from a user input and we want to make sure the user will not run dangerous commands like `shutdown`. Some command, such as `BLPOP`, are marked with `NO_SCRIPT` flag, which means they will not be allowed on script mode. Those commands are marked with `NO_SCRIPT` just because they are blocking commands and not because they are dangerous. Now that we can run blocking commands on RM_Call, there is no real reason not to allow such commands on script mode. The underline problem is that the `NO_SCRIPT` flag is abused to also mark some of the blocking commands (notice that those commands know not to block the client if it is not allowed to do so, and have a fallback logic to such cases. So even if those commands were not marked with `NO_SCRIPT` flag, it would not harm Redis, and today we can already run those commands within multi exec). In addition, not all blocking commands are marked with `NO_SCRIPT` flag, for example `blmpop` are not marked and can run from within a script. Those facts shows that there are some ambiguity about the meaning of the `NO_SCRIPT` flag, and its not fully clear where it should be use. The PR suggest that blocking commands should not be marked with `NO_SCRIPT` flag, those commands should handle `CLIENT_DENY_BLOCKING` flag and only block when it's safe (like they already does today). To achieve that, the PR removes the `NO_SCRIPT` flag from the following commands: * `blmove` * `blpop` * `brpop` * `brpoplpush` * `bzpopmax` * `bzpopmin` * `wait` This might be considered a breaking change as now, on scripts, instead of getting `command is not allowed from script` error, the user will get some fallback behavior base on the command implementation. That said, the change matches the behavior of scripts and multi exec with respect to those commands and allow running them on `RM_Call` even when script mode is used. ### Additional RedisModule API and changes * `RM_BlockClientSetPrivateData` - Set private data on the blocked client without the need to unblock the client. This allows up to set the promise CallReply as the private data of the blocked client and abort it if the client gets disconnected. * `RM_BlockClientGetPrivateData` - Return the current private data set on a blocked client. We need it so we will have access to this private data on the disconnect callback. * On RM_Call, the returned reply will be added to the auto memory context only if auto memory is enabled, this allows us to keep the call reply for longer time then the context lifetime and does not force an unneeded borrow relationship between the CallReply and the RedisModuleContext.
-