- 21 Aug, 2022 1 commit
-
-
yourtree authored
Till now Redis officially supported tuning it via environment variable see #1074. But we had other requests to allow changing it at runtime, see #799, and #11041. Note that `strcoll()` is used as Lua comparison function and also for comparison of certain string objects in Redis, which leads to a problem that, in different regions, for some characters, the result may be different. Below is an example. ``` 127.0.0.1:6333> SORT test alpha 1) "<" 2) ">" 3) "," 4) "*" 127.0.0.1:6333> CONFIG GET locale-collate 1) "locale-collate" 2) "" 127.0.0.1:6333> CONFIG SET locale-collate 1 (error) ERR CONFIG SET failed (possibly related to argument 'locale') 127.0.0.1:6333> CONFIG SET locale-collate C OK 127.0.0.1:6333> SORT test alpha 1) "*" 2) "," 3) "<" 4) ">" ``` That will cause accidental code compatibility issues for Lua scripts and some Redis commands. This commit creates a new config parameter to control the local environment which only affects `Collate` category. Above shows how it affects `SORT` command, and below shows the influence on Lua scripts. ``` 127.0.0.1:6333> CONFIG GET locale-collate 1) " locale-collate" 2) "C" 127.0.0.1:6333> EVAL "return ',' < '*'" 0 (nil) 127.0.0.1:6333> CONFIG SET locale-collate "" OK 127.0.0.1:6333> EVAL "return ',' < '*'" 0 (integer) 1 ``` Co-authored-by:
calvincjli <calvincjli@tencent.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 18 Aug, 2022 2 commits
-
-
guybe7 authored
This PR makes sure that "name" is unique for all arguments in the same level (i.e. all args of a command and all args within a block/oneof). This means several argument with identical meaning can be referred to together, but also if someone needs to refer to a specific one, they can use its full path. In addition, the "display_text" field has been added, to be used by redis.io in order to render the syntax of the command (for the vast majority it is identical to "name" but sometimes we want to use a different string that is not "name") The "display" field is exposed via COMMAND DOCS and will be present for every argument, except "oneof" and "block" (which are container arguments) Other changes: 1. Make sure we do not have any container arguments ("oneof" or "block") that contain less than two sub-args (otherwise it doesn't make sense) 2. migrate.json: both AUTH and AUTH2 should not be "optional" 3. arg names cannot contain underscores, and force the usage of hyphens (most of these were a result of the script that generated the initial json files from redis.io commands.json).
-
Meir Shpilraien (Spielrein) authored
Fix replication inconsistency on modules that uses key space notifications. ### The Problem In general, key space notifications are invoked after the command logic was executed (this is not always the case, we will discuss later about specific command that do not follow this rules). For example, the `set x 1` will trigger a `set` notification that will be invoked after the `set` logic was performed, so if the notification logic will try to fetch `x`, it will see the new data that was written. Consider the scenario on which the notification logic performs some write commands. for example, the notification logic increase some counter, `incr x{counter}`, indicating how many times `x` was changed. The logical order by which the logic was executed is has follow: ``` set x 1 incr x{counter} ``` The issue is that the `set x 1` command is added to the replication buffer at the end of the command invocation (specifically after the key space notification logic was invoked and performed the `incr` command). The replication/aof sees the commands in the wrong order: ``` incr x{counter} set x 1 ``` In this specific example the order is less important. But if, for example, the notification would have deleted `x` then we would end up with primary-replica inconsistency. ### The Solution Put the command that cause the notification in its rightful place. In the above example, the `set x 1` command logic was executed before the notification logic, so it should be added to the replication buffer before the commands that is invoked by the notification logic. To achieve this, without a major code refactoring, we save a placeholder in the replication buffer, when finishing invoking the command logic we check if the command need to be replicated, and if it does, we use the placeholder to add it to the replication buffer instead of appending it to the end. To be efficient and not allocating memory on each command to save the placeholder, the replication buffer array was modified to reuse memory (instead of allocating it each time we want to replicate commands). Also, to avoid saving a placeholder when not needed, we do it only for WRITE or MAY_REPLICATE commands. #### Additional Fixes * Expire and Eviction notifications: * Expire/Eviction logical order was to first perform the Expire/Eviction and then the notification logic. The replication buffer got this in the other way around (first notification effect and then the `del` command). The PR fixes this issue. * The notification effect and the `del` command was not wrap with `multi-exec` (if needed). The PR also fix this issue. * SPOP command: * On spop, the `spop` notification was fired before the command logic was executed. The change in this PR would have cause the replication order to be change (first `spop` command and then notification `logic`) although the logical order is first the notification logic and then the `spop` logic. The right fix would have been to move the notification to be fired after the command was executed (like all the other commands), but this can be considered a breaking change. To overcome this, the PR keeps the current behavior and changes the `spop` code to keep the right logical order when pushing commands to the replication buffer. Another PR will follow to fix the SPOP properly and match it to the other command (we split it to 2 separate PR's so it will be easy to cherry-pick this PR to 7.0 if we chose to). #### Unhanded Known Limitations * key miss event: * On key miss event, if a module performed some write command on the event (using `RM_Call`), the `dirty` counter would increase and the read command that cause the key miss event would be replicated to the replication and aof. This problem can also happened on a write command that open some keys but eventually decides not to perform any action. We decided not to handle this problem on this PR because the solution is complex and will cause additional risks in case we will want to cherry-pick this PR. We should decide if we want to handle it in future PR's. For now, modules writers is advice not to perform any write commands on key miss event. #### Testing * We already have tests to cover cases where a notification is invoking write commands that are also added to the replication buffer, the tests was modified to verify that the replica gets the command in the correct logical order. * Test was added to verify that `spop` behavior was kept unchanged. * Test was added to verify key miss event behave as expected. * Test was added to verify the changes do not break lazy expiration. #### Additional Changes * `propagateNow` function can accept a special dbid, -1, indicating not to replicate `select`. We use this to replicate `multi/exec` on `propagatePendingCommands` function. The side effect of this change is that now the `select` command will appear inside the `multi/exec` block on the replication stream (instead of outside of the `multi/exec` block). Tests was modified to match this new behavior.
-
- 15 Aug, 2022 1 commit
-
-
Oran Agra authored
The initial module format introduced in 4.0 RC1 and was changed in RC2 The initial function format introduced in 7.0 RC1 and changed in RC3
-
- 04 Aug, 2022 1 commit
-
-
filipe oliveira authored
Avoid the sdslen() on shared.crlf given we know its size beforehand. Improve ~3-4% of cpu cycles to lrange logic (#10987) * Avoid the sdslen() on shared.crlf given we know its size beforehand * Removed shared.crlf from sharedObjects
-
- 03 Aug, 2022 1 commit
-
-
Moti Cohen authored
Fixing few macros that doesn't follows most basic safety conventions which is wrapping any usage of passed variable with parentheses and if written more than one command, then wrap it with do-while(0) (or parentheses).
-
- 26 Jul, 2022 1 commit
-
-
Binbin authored
The reason we do this is because in #11036, we added error log message when failing to open RDB file for reading. In loadDdataFromDisk we call rdbLoad and also check errno, now the logging corrupts errno (reported in alpine daily). It is not safe to rely on errno as we do today, so we change the return value of rdbLoad function to enums, like we have when loading an AOF.
-
- 19 Jul, 2022 1 commit
-
-
Binbin authored
Following #10996, it forgot to modify RM_StringCompare in module.c Modified RM_StringCompare, compareStringObjectsWithFlags, compareStringObjects and collateStringObjects.
-
- 07 Jul, 2022 1 commit
-
-
Binbin authored
* Fix some outdated comments and some typo
-
- 06 Jul, 2022 1 commit
-
-
Binbin authored
We already have `pubsub_channels` and `pubsub_patterns` in INFO stats, now add `pubsubshard_channels` (symmetry). Sharded pubsub was added in #8621
-
- 04 Jul, 2022 2 commits
-
-
Harkrishn Patro authored
## Issue During the MULTI/EXEC flow, each command gets queued until the `EXEC` command is received and during this phase on every command queue, a `realloc` is being invoked. This could be expensive based on the realloc behavior (if copy to a new memory location). ## Solution In order to reduce the no. of syscall, couple of optimization I've used. 1. By default, reserve memory for atleast two commands. `MULTI/EXEC` for a single command doesn't have any significance. Hence, I believe customer wouldn't use it. 2. For further reservation, increase the memory allocation in exponent growth (power of 2). This reduces the no. of `realloc` call from `N` to `log(N)` times. ## Other changes: * Include multi exec queued command array in client memory consumption calculation (affects client eviction too)
-
Harkrishn Patro authored
Account sharded pubsub channels memory consumption in client memory usage computation to accurately evict client based on the set threshold for `maxmemory-clients`.
-
- 30 Jun, 2022 1 commit
-
-
Binbin authored
This was harmless because we marked the parent command with SENTINEL flag. So the populateCommandTable was ok. And we also don't show the flag (SENTINEL and ONLY-SENTNEL) in COMMAND INFO. In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL flags check when populating the sub-commands. so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
-
- 26 Jun, 2022 1 commit
-
-
Viktor Söderqvist authored
Adding Module APIs to let the module read and set the client name of an arbitrary connection.
-
- 12 Jun, 2022 1 commit
-
-
Binbin authored
The SET and BITFIELD command were added `get_keys_function` in #10148, causing them to be wrongly marked movablekeys in `populateCommandMovableKeys`. This was an unintended side effect introduced in #10148 (7.0 RC1) which could cause some clients an extra round trip for these commands in cluster mode. Since we define movablekeys as a way to determine if the legacy range [first, last, step] doesn't find all keys, then we need a completely different approach. The right approach should be to check if the legacy range covers all key-specs, and if none of the key-specs have the INCOMPLETE flag. This way, we don't need to look at getkeys_proc of VARIABLE_FLAG at all. Probably with the exception of modules, who may still not be using key-specs. In this PR, we removed `populateCommandMovableKeys` and put its logic in `populateCommandLegacyRangeSpec`. In order to properly serve both old and new modules, we must probably keep relying CMD_MODULE_GETKEYS, but do that only for modules that don't declare key-specs. For ones that do, we need to take the same approach we take with native redis commands. This approach was proposed by Oran. Fixes #10833 Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 06 Jun, 2022 1 commit
-
-
DarrenJiang13 authored
Split instantaneous_repl_total_kbps to instantaneous_input_repl_kbps and instantaneous_output_repl_kbps. (#10810) A supplement to https://github.com/redis/redis/pull/10062 Split `instantaneous_repl_total_kbps` to `instantaneous_input_repl_kbps` and `instantaneous_output_repl_kbps`. ## Work: This PR: - delete 1 info field: - `instantaneous_repl_total_kbps` - add 2 info fields: - `instantaneous_input_repl_kbps / instantaneous_output_repl_kbps` ## Result: - master ``` total_net_input_bytes:26633673 total_net_output_bytes:21716596 total_net_repl_input_bytes:0 total_net_repl_output_bytes:18433052 instantaneous_input_kbps:0.02 instantaneous_output_kbps:0.00 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 ``` - slave ``` total_net_input_bytes:18433212 total_net_output_bytes:94790 total_net_repl_input_bytes:18433052 total_net_repl_output_bytes:0 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.05 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 ```
-
- 03 Jun, 2022 1 commit
-
-
Madelyn Olson authored
* Update time independent string compare to use hash length
-
- 01 Jun, 2022 3 commits
-
-
Oran Agra authored
The important part is that read-only scripts (not just EVAL_RO and FCALL_RO, but also ones with `no-writes` executed by normal EVAL or FCALL), will now be permitted to run during CLIENT PAUSE WRITE (unlike before where only the _RO commands would be processed). Other than that, some errors like OOM, READONLY, MASTERDOWN are now handled by processCommand, rather than the command itself affects the error string (and even error code in some cases), and command stats. Besides that, now the `may-replicate` commands, PFCOUNT and PUBLISH, will be considered `write` commands in scripts and will be blocked in all read-only scripts just like other write commands. They'll also be blocked in EVAL_RO (i.e. even for scripts without the `no-writes` shebang flag. This commit also hides the `may_replicate` flag from the COMMAND command output. this is a **breaking change**. background about may_replicate: We don't want to expose a no-may-replicate flag or alike to scripts, since we consider the may-replicate thing an internal concern of redis, that we may some day get rid of. In fact, the may-replicate flag was initially introduced to flag EVAL: since we didn't know what it's gonna do ahead of execution, before function-flags existed). PUBLISH and PFCOUNT, both of which because they have side effects which may some day be fixed differently. code changes: The changes in eval.c are mostly code re-ordering: - evalCalcFunctionName is extracted out of evalGenericCommand - evalExtractShebangFlags is extracted luaCreateFunction - evalGetCommandFlags is new code
-
Oran Agra authored
Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check bug, and new RM_Call checks. (#10786) * Fix broken protocol when redis can't persist to RDB (general commands, not modules), excessive newline. regression of #10372 (7.0 RC3) * Fix broken protocol when Redis can't persist to AOF (modules and scripts), missing newline. * Fix bug in OOM check of EVAL scripts called from RM_Call. set the cached OOM state for scripts before executing module commands too, so that it can serve scripts that are executed by modules. i.e. in the past EVAL executed by RM_Call could have either falsely fail or falsely succeeded because of a wrong cached OOM state flag. * Fix bugs with RM_Yield: 1. SHUTDOWN should only accept the NOSAVE mode 2. Avoid eviction during yield command processing. 3. Avoid processing master client commands while yielding from another client * Add new two more checks to RM_Call script mode. 1. READONLY You can't write against a read only replica 2. MASTERDOWN Link with MASTER is down and `replica-serve-stale-data` is set to `no` * Add new RM_Call flag to let redis automatically refuse `deny-oom` commands while over the memory limit. * Add tests to cover various errors from Scripts, Modules, Modules calling scripts, and Modules calling commands in script mode. Add tests: * Looks like the MISCONF error was completely uncovered by the tests, add tests for it, including from scripts, and modules * Add tests for NOREPLICAS from scripts * Add tests for the various errors in module RM_Call, including RM_Call that calls EVAL, and RM_call in "eval mode". that includes: NOREPLICAS, READONLY, MASTERDOWN, MISCONF
-
filipe oliveira authored
Move the client flags to a more cache friendly position within the client struct we regain the lost 2% of CPU cycles since v6.2 ( from 630532.57 to 647449.80 ops/sec ). These are due to higher rate of calls to getClientType due to changes in #9166 and #10020
-
- 31 May, 2022 2 commits
-
-
DarrenJiang13 authored
The amount of `server.stat_net_output_bytes/server.stat_net_input_bytes` is actually the sum of replication flow and users' data flow. It may cause confusions like this: "Why does my server get such a large output_bytes while I am doing nothing? ". After discussions and revisions, now here is the change about what this PR brings (final version before merge): - 2 server variables to count the network bytes during replication, including fullsync and propagate bytes. - `server.stat_net_repl_output_bytes`/`server.stat_net_repl_input_bytes` - 3 info fields to print the input and output of repl bytes and instantaneous value of total repl bytes. - `total_net_repl_input_bytes` / `total_net_repl_output_bytes` - `instantaneous_repl_total_kbps` - 1 new API `rioCheckType()` to check the type of rio. So we can use this to distinguish between diskless and diskbased replication - 2 new counting items to keep network statistics consistent between master and slave - rdb portion during diskless replica. in `rdbLoadProgressCallback()` - first line of the full sync payload. in `readSyncBulkPayload()` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Harkrishn Patro authored
To easily distinguish between sharded channel message and a global channel message, introducing `smessage` (instead of `message`) as message bulk for sharded channel publish message. This is gonna be a breaking change in 7.0.1! Background: Sharded pubsub introduced in redis 7.0, but after the release we quickly realized that the fact that it's problematic that the client can't distinguish between normal (global) pubsub messages and sharded ones. This is important because the same connection can subscribe to both, but messages sent to one pubsub system are not propagated to the other (they're completely separate), so if one connection is used to subscribe to both, we need to assist the client library to know which message it got so it can forward it to the correct callback.
-
- 29 May, 2022 1 commit
-
-
Binbin authored
The ret value should be AOF_OK instead of C_OK. AOF_OK and C_OK are both 0, so this is just a cleanup. Also updated some outdated comments.
-
- 22 May, 2022 1 commit
-
-
Binbin authored
Remove some dead code in object.c, ziplist is no longer used in 7.0 Some backgrounds: zipmap - hash: replaced by ziplist in #285 ziplist - hash: replaced by listpack in #8887 ziplist - zset: replaced by listpack in #9366 ziplist - list: replaced by quicklist (listpack) in #2143 / #9740 Moved the location of ziplist.h in the server.c
-
- 15 May, 2022 1 commit
-
-
Qu Chen authored
sometimes it is using `scriptIsRunning()` and other times it is using `server.in_script`. We should use the `scriptIsRunning()` method consistently throughout the code base. Removed server.in_script sine it's no longer used / needed.
-
- 12 May, 2022 1 commit
-
-
yoav-steinberg authored
Fix possible regression around TLS config changes. Add VOLATILE_CONFIG flag for volatile configurations. (#10713) This fixes a possible regression in Redis 7.0.0, in which doing CONFIG SET on a TLS config would not reload the configuration in case the new config is the same file as before. A volatile configuration is a configuration value which is a reference to the configuration data and not the configuration data itself. In such a case Redis doesn't know if the config data changed under the hood and can't assume a change happens only when the config value changes. Therefore it needs to be applied even when setting a config value to the same value as it was before.
-
- 10 May, 2022 1 commit
-
-
guybe7 authored
Fix #10552 We no longer piggyback getkeys_proc to hold the RedisModuleCommand struct, when exists Others: Use `doesCommandHaveKeys` in `RM_GetCommandKeysWithFlags` and `getKeysSubcommandImpl`. It causes a very minor behavioral change in commands that don't have actual keys, but have a spec with `CMD_KEY_NOT_KEY`. For example, before this command `COMMAND GETKEYS SPUBLISH` would return `Invalid arguments specified for command` but not it returns `The command has no key arguments`
-
- 08 May, 2022 1 commit
-
-
David CARLIER authored
also fixing already defined constants build warning while at it. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 26 Apr, 2022 5 commits
-
-
chenyang8094 authored
Fix bug when AOF enabled after startup. put the new incr file in the manifest only when AOFRW is done. (#10616) Changes: - When AOF is enabled **after** startup, the data accumulated during `AOF_WAIT_REWRITE` will only be stored in a temp INCR AOF file. Only after the first AOFRW is successful, we will add it to manifest file. Before this fix, the manifest referred to the temp file which could cause a restart during that time to load it without it's base. - Add `aof_rewrites_consecutive_failures` info field for aofrw limiting implementation. Now we can guarantee that these behaviors of MP-AOF are the same as before (past redis releases): - When AOF is enabled after startup, the data accumulated during `AOF_WAIT_REWRITE` will only be stored in a visible place. Only after the first AOFRW is successful, we will add it to manifest file. - When disable AOF, we did not delete the AOF file in the past so there's no need to change that behavior now (yet). - When toggling AOF off and then on (could be as part of a full-sync), a crash or restart before the first rewrite is completed, would result with the previous version being loaded (might not be right thing, but that's what we always had).
-
Eduardo Semprebon authored
The SHUTDOWN command has various flags to change it's default behavior, but in some cases establishing a connection to redis is complicated and it's easier for the management software to use signals. however, so far the signals could only trigger the default shutdown behavior. Here we introduce the option to control shutdown arguments for SIGTERM and SIGINT. New config options: `shutdown-on-sigint [nosave | save] [now] [force]` `shutdown-on-sigterm [nosave | save] [now] [force]` Implementation: Support MULTI_ARG_CONFIG on createEnumConfig to support multiple enums to be applied as bit flags. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Binbin authored
Missing a typeof, we will get errors like this: - multiple definition of `replicationErrorBehavior' - ld: error: duplicate symbol: replicationErrorBehavior Introduced in #10504
-
Madelyn Olson authored
* Till now, replicas that were unable to persist, would still execute the commands they got from the master, now they'll panic by default, and we add a new `replica-ignore-disk-errors` config to change that. * Till now, when a command failed on a replica or AOF-loading, it only logged a warning and a stat, we add a new `propagation-error-behavior` config to allow panicking in that state (may become the default one day) Note that commands that fail on the replica can either indicate a bug that could cause data inconsistency between the replica and the master, or they could be in some cases (specifically in previous versions), a result of a command (e.g. EVAL) that failed on the master, but still had to be propagated to fail on the replica as well.
-
Madelyn Olson authored
Adds the `allow-cross-slot-keys` flag to Eval scripts and Functions to allow scripts to access keys from multiple slots. The default behavior is now that they are not allowed to do that (unlike before). This is a breaking change for 7.0 release candidates (to be part of 7.0.0), but not for previous redis releases since EVAL without shebang isn't doing this check. Note that the check is done on both the keys declared by the EVAL / FCALL command arguments, and also the ones used by the script when making a `redis.call`. A note about the implementation, there seems to have been some confusion about allowing access to non local keys. I thought I missed something in our wider conversation, but Redis scripts do block access to non-local keys. So the issue was just about cross slots being accessed.
-
- 25 Apr, 2022 2 commits
-
-
Binbin authored
minor cleanup for recent changes.
-
guybe7 authored
1. Disk error and slave count checks didn't flag the transactions or counted correctly in command stats (regression from #10372 , 7.0 RC3) 2. RM_Call will reply the same way Redis does, in case of non-exisitng command or arity error 3. RM_WrongArtiy will consider the full command name 4. Use lowercase 'u' in "unknonw subcommand" (to align with "unknown command") Followup work of #10127
-
- 20 Apr, 2022 2 commits
-
-
Oran Agra authored
This PR unifies all the places that test if the current client is the master client or AOF client, and uses a method to test that on all of these. Other than some refactoring, these are the actual implications: - Replicas **don't** ignore disk error when processing commands not coming from their master. **This is important for PING to be used for health check of replicas** - SETRANGE, APPEND, SETBIT, BITFIELD don't do proto_max_bulk_len check for AOF - RM_Call in SCRIPT_MODE ignores disk error when coming from master / AOF - RM_Call in cluster mode ignores slot check when processing AOF - Scripts ignore disk error when processing AOF - Scripts **don't** ignore disk error on a replica, if the command comes from clients other than the master - SCRIPT KILL won't kill script coming from AOF - Scripts **don't** skip OOM check on replica if the command comes from clients other than the master Note that Script, AOF, and module clients don't reach processCommand, which is why some of the changes don't actually have any implications. Note, reverting the change done to processCommand in 2f4240b9 should be dead code due to the above mentioned fact.
-
David CARLIER authored
Add a configuration option to attach an operating system-specific identifier to Redis sockets, supporting advanced network configurations using iptables (Linux) or ipfw (FreeBSD).
-
- 17 Apr, 2022 1 commit
-
-
guybe7 authored
since PUBLISH and SPUBLISH use different dictionaries for channels and clients, and we already have an API for PUBLISH, it only makes sense to have one for SPUBLISH Add test coverage and unifying some test infrastructure.
-
- 13 Apr, 2022 2 commits
-
-
Luke Palmer authored
Add an optional keyspace event when new keys are added to the db. This is useful for applications where clients need to be aware of the redis keyspace. Such an application can SCAN once at startup and then listen for "new" events (plus others associated with DEL, RENAME, etc).
-
guybe7 authored
Apparently, some modules can afford deprecating command arguments (something that was never done in Redis, AFAIK), so in order to represent this piece of information, we added the `deprecated_since` field to redisCommandArg (in symmetry to the already existing `since` field). This commit adds `const char *deprecated_since` to `RedisModuleCommandArg`, which is technically a breaking change, but since 7.0 was not released yet, we decided to let it slide
-