- 04 Jan, 2022 5 commits
-
-
Matthieu MOREL authored
This sets up dependabot to check weekly updates for pip and github-actions dependencies. If it finds an update it will create a PR to update the dependency. More information can be found here It includes the update of: * vmactions/freebsd-vm from 0.1.4 to 0.1.5 * codespell from 2.0.0 to 2.1.0 Also includes spelling fixes found by the latest version of codespell. Includes a dedicated .codespell folder so dependabot can read a requirements.txt file and every files dedicated to codespell can be grouped in the same place Co-Authored-By:
Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-Authored-By:
MOREL Matthieu <matthieu.morel@cnp.fr>
-
Binbin authored
In monitor/pubsub mode, if the server closes the connection, for example, use `CLIENT KILL`, redis-cli will exit directly without printing any error messages. This commit ensures that redis-cli will try to print the error messages before exiting. Also there is a minor cleanup for restart, see the example below. before: ``` 127.0.0.1:6379> monitor OK [root@ redis]# 127.0.0.1:6379> subscribe channel Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "channel" 3) (integer) 1 [root@ redis]# 127.0.0.1:6379> restart 127.0.0.1:6379> get keyUse 'restart' only in Lua debugging mode. (nil) ``` after: ``` 127.0.0.1:6379> monitor OK Error: Server closed the connection [root@ redis]# 127.0.0.1:6379> subscribe channel Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "channel" 3) (integer) 1 Error: Server closed the connection [root@ redis]# 127.0.0.1:6379> restart Use 'restart' only in Lua debugging mode. ```
-
王辉 authored
Older version of GNU Make (<4.3) required quoting of number signs (#) to avoid them being treated as a comment. Newer versions will treat this quote as a literal. This issue and a proposed solution is discussed here: https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
guybe7 authored
Creating fork (or even a foreground SAVE) during a transaction breaks the atomicity of the transaction. In addition to that, it could mess up the propagated transaction to the AOF file. This change blocks SAVE, PSYNC, SYNC and SHUTDOWN from being executed inside MULTI-EXEC. It does that by adding a command flag, so that modules can flag their commands with that flag too. Besides it changes BGSAVE, BGREWRITEAOF, and CONFIG SET appendonly, to turn the scheduled flag instead of forking righ taway. Other changes: * expose `protected`, `no-async-loading`, and `no_multi` flags in COMMAND command * add a test to validate propagation of FLUSHALL inside a transaction. * add a test to validate how CONFIG SET that errors reacts in a transaction Co-authored-by:
Oran Agra <oran@redislabs.com>
-
zhaozhao.zz authored
This would mean that the effects of `CONFIG SET maxmemory` may not be visible once the command returns. That could anyway happen since incremental eviction was added in redis 6.2 (see #7653) We do this to fix one of the propagation bugs about eviction see #9890 and #10014.
-
- 03 Jan, 2022 4 commits
-
-
chenyang8094 authored
Implement Multi-Part AOF mechanism to avoid overheads during AOFRW. Introducing a folder with multiple AOF files tracked by a manifest file. The main issues with the the original AOFRW mechanism are: * buffering of commands that are processed during rewrite (consuming a lot of RAM) * freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it. * double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files) The main modifications of this PR: 1. Remove the AOF rewrite buffer and related code. 2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type, it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the incremental commands since the last AOFRW. 3. Use a AOF manifest file to record and manage these AOF files mentioned above. 4. The original configuration of `appendfilename` will be the base part of the new file name, for example: `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof` 5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename` 6. Remove the `aof_rewrite_buffer_length` field in info. 7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs. It also gives users the opportunity to preserve the history AOFs. just for testing use now. 8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now), we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately. 9. Support upgrade (load) data from old version redis. 10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and manifest file will be placed in this directory. 11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if `aof-load-truncated` is enabled. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Meir Shpilraien (Spielrein) authored
OOM Error did not raise on functions due to a bug. Added test to verify the fix.
-
Madelyn Olson authored
Implement the ability for cluster nodes to advertise their location with extension messages.
-
Harkrishn Patro authored
This commit implements a sharded pubsub implementation based off of shard channels. Co-authored-by:
Harkrishn Patro <harkrisp@amazon.com> Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 02 Jan, 2022 5 commits
-
-
Binbin authored
This commit adds DUMP RESTORES tests for the -x and -X options. I wanted to add it in #9980 which introduce the -X option, but back then i failed due to some errors (related to redis-cli call).
-
yoav-steinberg authored
Since #9166 we have an assertion here to make sure replica clients don't write anything to their buffer. But in reality a replica may attempt write data to it's buffer simply by sending a command on the replication link. This command in most cases will be rejected since #8868 but it'll still generate an error. Actually the only valid command to send on a replication link is 'REPCONF ACK` which generates no response. We want to keep the design so that replicas can send commands but we need to avoid any situation where we start putting data in their response buffers, especially since they aren't used anymore. This PR makes sure to disconnect a rogue client which generated a write on the replication link that cause something to be written to the response buffer. To recreate the bug this fixes simply connect via telnet to a redis server and write sync\r\n wait for the the payload to be written and then write any command (valid or invalid), such as ping\r\n on the telnet connection. It'll crash the server.
-
Joey from AWS authored
Report slot to keys map size in MEMORY STATS in cluster mode Report dictMetadataSize in MEMORY USAGE command as well
-
Viktor Söderqvist authored
To avoid data loss, this commit adds a grace period for lagging replicas to catch up the replication offset. Done: * Wait for replicas when shutdown is triggered by SIGTERM and SIGINT. * Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients to call SHUTDOWN in parallel. Note that they don't expect a response unless an error happens and shutdown is aborted. * Log warning for each replica lagging behind when finishing shutdown. * CLIENT_PAUSE_WRITE while waiting for replicas. * Configurable grace period 'shutdown-timeout' in seconds (default 10). * New flags for the SHUTDOWN command: - NOW disables the grace period for lagging replicas. - FORCE ignores errors writing the RDB or AOF files which would normally prevent a shutdown. - ABORT cancels ongoing shutdown. Can't be combined with other flags. * New field in the output of the INFO command: 'shutdown_in_milliseconds'. The value is the remaining maximum time to wait for lagging replicas before finishing the shutdown. This field is present in the Server section **only** during shutdown. Not directly related: * When shutting down, if there is an AOF saving child, it is killed **even** if AOF is disabled. This can happen if BGREWRITEAOF is used when AOF is off. * Client pause now has end time and type (WRITE or ALL) per purpose. The different pause purposes are *CLIENT PAUSE command*, *failover* and *shutdown*. If clients are unpaused for one purpose, it doesn't affect client pause for other purposes. For example, the CLIENT UNPAUSE command doesn't affect client pause initiated by the failover or shutdown procedures. A completed failover or a failed shutdown doesn't unpause clients paused by the CLIENT PAUSE command. Notes: * DEBUG RESTART doesn't wait for replicas. * We already have a warning logged when a replica disconnects. This means that if any replica connection is lost during the shutdown, it is either logged as disconnected or as lagging at the time of exit. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
yoav-steinberg authored
This is needed in order to ease the deployment of functions for ephemeral cases, where user needs to spin up a server with functions pre-loaded. #### Details: * Added `--functions-rdb` option to _redis-cli_. * Functions only rdb via `REPLCONF rdb-filter-only functions`. This is a placeholder for a space separated inclusion filter for the RDB. In the future can be `REPLCONF rdb-filter-only "functions db:3 key-patten:user*"` and a complementing `rdb-filter-exclude` `REPLCONF` can also be added. * Handle "slave requirements" specification to RDB saving code so we can use the same RDB when different slaves express the same requirements (like functions-only) and not share the RDB when their requirements differ. This is currently just a flags `int`, but can be extended to a more complex structure with various filter fields. * make sure to support filters only in diskless replication mode (not to override the persistence file), we do that by forcing diskless (even if disabled by config) other changes: * some refactoring in rdb.c (extract portion of a big function to a sub-function) * rdb_key_save_delay used in AOFRW too * sendChildInfo takes the number of updated keys (incremental, rather than absolute) Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 01 Jan, 2022 1 commit
-
-
sundb authored
This pr is mainly to solve the problem that redis process cannot be exited normally, due to changes in #10003. When a test uses the `key-load-delay` config to delay loading, but does not reset it at the end of the test, will lead to server wait for the loading to reach the event loop (once in 2mb) before actually shutting down.
-
- 31 Dec, 2021 2 commits
-
-
Madelyn Olson authored
-
sundb authored
Now if redis is still loading when we receive sigterm, we will wait for the loading to reach the event loop (once in 2mb) before actually shutting down. See #10003. This change caused valgrind CI to fail. See https://github.com/redis/redis/runs/4662901673?check_suite_focus=true This pr is mainly to solve the problem that redis process cannot be exited normally. When the master is disconnected, if repl is processing diskless loading and using `connRead` to read data from master, it may enter an infinite retry state, which does not handle `connRead` returning 0(master connection disconnected).
-
- 30 Dec, 2021 3 commits
-
-
Viktor Söderqvist authored
These exist for quite some time, and are no longer experimental
-
Binbin authored
There are two changes in this commit: 1. Add -X option to redis-cli. Currently `-x` can only be used to provide the last argument, so you can do `redis-cli dump keyname > key.dump`, and then do `redis-cli -x restore keyname 0 < key.dump`. But what if you want to add the replace argument (which comes last?). oran suggested adding such usage: `redis-cli -X <tag> restore keyname <tag> replace < key.dump` i.e. you're able to provide a string in the arguments that's gonna be substituted with the content from stdin. Note that the tag name should not conflict with others non-replaced args. And the -x and -X options are conflicting. Some usages: ``` [root]# echo mypasswd | src/redis-cli -X passwd_tag mset username myname password passwd_tag OK [root]# echo username > username.txt [root]# head -c -1 username.txt | src/redis-cli -X name_tag mget name_tag password 1) "myname" 2) "mypasswd\n" ``` 2. Handle the combination of both `-x` and `--cluster` or `-X` and `--cluster` Extend the broadcast option to receive the last arg or <tag> arg from the stdin. Now we can use `redis-cli -x --cluster call <host>:<port> cmd`, or `redis-cli -X <tag> --cluster call <host>:<port> cmd <tag>`. (support part of #9899)
-
Viktor Söderqvist authored
Add blank before lists, so that they will be rendered as lists. Commands RM_GetCommand and RM_CreateSubcommand were added in #9656.
-
- 29 Dec, 2021 1 commit
-
-
Itamar Haber authored
Add missing information about commands, mainly from reviewing redis-doc and removing the metadata from it (https://github.com/redis/redis-doc/pull/1722) * Reintroduces CLUSTER S****S (supported by Redis) but missing from the JSON / docs (related? #9675). Note that without that json file, the command won't work (breaking change) * Adds the `replicas` argument (exists in Redis) to `CLIENT KILL`. * Adds `history` entries to several commands based on redis-doc's man pages. * Adds `since` to applicable command arguments based on `history` (this basically makes some of `history` redundant - perhaps at a later stage). * Uses proper semantic versioning in all version references. * Also removes `geoencodeCommand` and `geodecodeCommand` header declarations per b96af595.
-
- 28 Dec, 2021 4 commits
-
-
Ozan Tezcan authored
-
perryitay authored
on the signal handler we are enabling server.shutdown_asap flag instead of just doing `exit()`, and then catch it on the whileBlockedCron() where we prepare for shutdown correctly. this is a more ore organized and safe termination, the old approach was missing these for example: 1. removal of the pidfile 2. shutdown event to modules
-
guybe7 authored
Preventing COFIG SET maxmemory from propagating is just the tip of the iceberg. Module that performs a write operation in a notification can cause any command to be propagated, based on server.dirty We need to come up with a better solution.
-
David CARLIER authored
Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
- 27 Dec, 2021 4 commits
-
-
chenyang8094 authored
It turns out that libc malloc can return an allocation of a different size on requests of the same size. this means that matching MEMORY USAGE of one key to another copy of the same data can fail. Solution: Keep running the test that calls MEMORY USAGE, but ignore the response. We do that by introducing a new utility function to get the memory usage, which always returns 1 when the allocator is not jemalloc. Other changes: Some formatting for datatype2.tcl Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Itamar Haber authored
Following #9656, this script generates a "commands.json" file from the output of the new COMMAND. The output of this script is used in redis/redis-doc#1714 and by redis/redis-io#259. This also converts a couple of rogue dashes (in 'key-specs' and 'multiple-token' flags) to underscores (continues #9959).
-
chenyang8094 authored
PR #9890 may have introduced a problem. There are tests that use MULTI-EXEC to make sure two BGSAVE / BGREWRITEAOF are executed together. But now it's not valid to run run commands that create a snapshot inside a transaction (gonna be blocked soon) This PR modifies the test not to rely on MULTI-EXEC. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
guybe7 authored
There's a race between testing DBSIZE and the thread starting. If the thread hadn't started by the time we checked DBISZE, no keys will have been evicted. The correct way is to check the evicted_keys stat.
-
- 26 Dec, 2021 3 commits
-
-
Binbin authored
`zslInsert` with a NAN score will crash the server. This one found by the `corrupt-dump-fuzzer`.
-
Meir Shpilraien (Spielrein) authored
Follow the conclusions to support Functions in redis cluster (#9899) Added 2 new FUNCTION sub-commands: 1. `FUNCTION DUMP` - dump a binary payload representation of all the functions. 2. `FUNCTION RESTORE <PAYLOAD> [FLUSH|APPEND|REPLACE]` - give the binary payload extracted using `FUNCTION DUMP`, restore all the functions on the given payload. Restore policy can be given to control how to handle existing functions (default is APPEND): * FLUSH: delete all existing functions. * APPEND: appends the restored functions to the existing functions. On collision, abort. * REPLACE: appends the restored functions to the existing functions. On collision, replace the old function with the new function. Modify `redis-cli --cluster add-node` to use `FUNCTION DUMP` to get existing functions from one of the nodes in the cluster, and `FUNCTION RESTORE` to load the same set of functions to the new node. `redis-cli` will execute this step before sending the `CLUSTER MEET` command to the new node. If `FUNCTION DUMP` returns an error, assume the current Redis version do not support functions and skip `FUNCTION RESTORE`. If `FUNCTION RESTORE` fails, abort and do not send the `CLUSTER MEET` command. If the new node already contains functions (before the `FUNCTION RESTORE` is sent), abort and do not add the node to the cluster. Test was added to verify `redis-cli --cluster add-node` works as expected.
-
Meir Shpilraien (Spielrein) authored
Use case insensitive string comparison for function names (like we do for commands and configs) In addition, add verification that the functions only use the following characters: [a-zA-Z0-9_]
-
- 23 Dec, 2021 1 commit
-
-
Yuta Hongo authored
Introduce `redis-cli --json` option. CSV doesn't support Map type, then parsing SLOWLOG or HMSET with multiple args are not helpful. By default the `--json` implies RESP3, which makes it much more useful, and a `-2` option was added to force RESP2. When `HELLO 3` fails, it prints a warning message (which can be silenced with `-2`). If a user passed `-3` explicitly, the non-interactive mode will also exit with error without running the command, while in interactive session it'll keep running after printing the warning. JSON output would be helpful to parse Redis replies with other tools like jq. ``` redis-cli --json slowlog get | jq [ [ 1, 1639677545, 322362, [ "HMSET", "dummy-key", "field1", "123,456,789... (152 more bytes)", "field2", "111,222,333... (140 more bytes)", "field3", "... (349 more arguments)" ], "127.0.0.1:49312", "" ] ] ```
-
- 22 Dec, 2021 6 commits
-
-
guybe7 authored
The mess: Some parts use alsoPropagate for late propagation, others using an immediate one (propagate()), causing edge cases, ugly/hacky code, and the tendency for bugs The basic idea is that all commands are propagated via alsoPropagate (i.e. added to a list) and the top-most call() is responsible for going over that list and actually propagating them (and wrapping them in MULTI/EXEC if there's more than one command). This is done in the new function, propagatePendingCommands. Callers to propagatePendingCommands: 1. top-most call() (we want all nested call()s to add to the also_propagate array and just the top-most one to propagate them) - via `afterCommand` 2. handleClientsBlockedOnKeys: it is out of call() context and it may propagate stuff - via `afterCommand`. 3. handleClientsBlockedOnKeys edge case: if the looked-up key is already expired, we will propagate the expire but will not unblock any client so `afterCommand` isn't called. in that case, we have to propagate the deletion explicitly. 4. cron stuff: active-expire and eviction may also propagate stuff 5. modules: the module API allows to propagate stuff from just about anywhere (timers, keyspace notifications, threads). I could have tried to catch all the out-of-call-context places but it seemed easier to handle it in one place: when we free the context. in the spirit of what was done in call(), only the top-most freeing of a module context may cause propagation. 6. modules: when using a thread-safe ctx it's not clear when/if the ctx will be freed. we do know that the module must lock the GIL before calling RM_Replicate/RM_Call so we propagate the pending commands when releasing the GIL. A "known limitation", which were actually a bug, was fixed because of this commit (see propagate.tcl): When using a mix of RM_Call with `!` and RM_Replicate, the command would propagate out-of-order: first all the commands from RM_Call, and then the ones from RM_Replicate Another thing worth mentioning is that if, in the past, a client would issue a MULTI/EXEC with just one write command the server would blindly propagate the MULTI/EXEC too, even though it's redundant. not anymore. This commit renames propagate() to propagateNow() in order to cause conflicts in pending PRs. propagatePendingCommands is the only caller of propagateNow, which is now a static, internal helper function. Optimizations: 1. alsoPropagate will not add stuff to also_propagate if there's no AOF and replicas 2. alsoPropagate reallocs also_propagagte exponentially, to save calls to memmove Bugfixes: 1. CONFIG SET can create evictions, sending notifications which can cause to dirty++ with modules. we need to prevent it from propagating to AOF/replicas 2. We need to set current_client in RM_Call. buggy scenario: - CONFIG SET maxmemory, eviction notifications, module hook calls RM_Call - assertion in lookupKey crashes, because current_client has CONFIG SET, which isn't CMD_WRITE 3. minor: in eviction, call propagateDeletion after notification, like active-expire and all commands (we always send a notification before propagating the command)
-
Oran Agra authored
issue started failing after #9878 was merged (made an exiting test more sensitive) looks like #9982 didn't help, tested this one and it seems to work better. this commit does two things: 1. reduce the extra delay i added earlier and instead add more keys, the effect no duration of replication is the same, but the intervals in which the server is responsive to the tcl client is higher. 2. improve the test infra to print context when assert_error fails.
-
Oran Agra authored
issue started failing after #9878 was merged (made an exiting test more sensitive)
-
Oran Agra authored
## background Till now CONFIG SET was blocked during loading. (In the not so distant past, GET was disallowed too) We recently (not released yet) added an async-loading mode, see #9323, and during that time it'll serve CONFIG SET and any other command. And now we realized (#9770) that some configs, and commands are dangerous during async-loading. ## changes * Allow most CONFIG SET during loading (both on async-loading and normal loading) * Allow CONFIG REWRITE and CONFIG RESETSTAT during loading * Block a few config during loading (`appendonly`, `repl-diskless-load`, and `dir`) * Block a few commands during loading (list below) ## the blocked commands: * SAVE - obviously we don't wanna start a foregreound save during loading 8-) * BGSAVE - we don't mind to schedule one, but we don't wanna fork now * BGREWRITEAOF - we don't mind to schedule one, but we don't wanna fork now * MODULE - we obviously don't wanna unload a module during replication / rdb loading (MODULE HELP and MODULE LIST are not blocked) * SYNC / PSYNC - we're in the middle of RDB loading from master, must not allow sync requests now. * REPLICAOF / SLAVEOF - we're in the middle of replicating, maybe it makes sense to let the user abort it, but he couldn't do that so far, i don't wanna take any risk of bugs due to odd state. * CLUSTER - only allow [HELP, SLOTS, NODES, INFO, MYID, LINKS, KEYSLOT, COUNTKEYSINSLOT, GETKEYSINSLOT, RESET, REPLICAS, COUNT_FAILURE_REPORTS], for others, preserve the status quo ## other fixes * processEventsWhileBlocked had an issue when being nested, this could happen with a busy script during async loading (new), but also in a busy script during AOF loading (old). this lead to a crash in the scenario described in #6988
-
zhugezy authored
If a test fails at `wait_for_blocked_clients_count` after the `PAUSE` command, It won't send `UNPAUSE` to server, leading to the server hanging until timeout, which is bad and hard to debug sometimes when developing. This PR tries to fix this. Timeout in `CLIENT PAUSE` shortened from 1e5 seconds(extremely long) to 50~100 seconds.
-
Hongcai Ren authored
when count is 0, return NULL
-
- 21 Dec, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
The issue with MAY_REPLICATE is that all automatic mechanisms to handle write commands will not work. This require have a special treatment for: * Not allow those commands to be executed on RO replica. * Allow those commands to be executed on RO replica from primary connection. * Allow those commands to be executed on the RO replica from AOF. By setting those commands as WRITE commands we are getting all those properties from Redis. Test was added to verify that those properties work as expected. In addition, rearrange when and where functions are flushed. Before this PR functions were flushed manually on `rdbLoadRio` and cleaned manually on failure. This contradicts the assumptions that functions are data and need to be created/deleted alongside with the data. A side effect of this, for example, `debug reload noflush` did not flush the data but did flush the functions, `debug loadaof` flush the data but not the functions. This PR move functions deletion into `emptyDb`. `emptyDb` (renamed to `emptyData`) will now accept an additional flag, `NOFUNCTIONS` which specifically indicate that we do not want to flush the functions (on all other cases, functions will be flushed). Used the new flag on FLUSHALL and FLUSHDB only! Tests were added to `debug reload` and `debug loadaof` to verify that functions behave the same as the data. Notice that because now functions will be deleted along side with the data we can not allow `CLUSTER RESET` to be called from within a function (it will cause the function to be released while running), this PR adds `NO_SCRIPT` flag to `CLUSTER RESET` so it will not be possible to be called from within a function. The other cluster commands are allowed from within a function (there are use-cases that uses `GETKEYSINSLOT` to iterate over all the keys on a given slot). Tests was added to verify `CLUSTER RESET` is denied from within a script. Another small change on this PR is that `RDBFLAGS_ALLOW_DUP` is also applicable on functions. When loading functions, if this flag is set, we will replace old functions with new ones on collisions.
-