- 14 Jan, 2022 1 commit
-
-
Peter Hickey authored
-
- 13 Jan, 2022 3 commits
-
-
Binbin authored
The dbs doesn't have any keys, `rdb-key-save-delay` config has no effect that cause the rewrite to complete. It was introduced in #10015.
-
Ozan Tezcan authored
Added RM_MonotonicMicroseconds(). Modules can use monotonic timestamp counter for measurements.
-
chenyang8094 authored
Force create a BASE file (use a foreground `rewriteAppendOnlyFile`) when redis starts from an empty data set and `appendonly` is yes. The reasoning is that normally, after redis is running for some time, and the AOF has gone though a few rewrites, there's always a base rdb file. and the scenario where the base file is missing, is kinda rare (happens only at empty startup), so this change normalizes it. But more importantly, there are or could be some complex modules that are started with some configuration, when they create persistence they write that configuration to RDB AUX fields, so that can can always know with which configuration the persistence file they're loading was created (could be critical). there is (was) one scenario in which they could load their persisted data, and that configuration was missing, and this change fixes it. Add a new module event: REDISMODULE_SUBEVENT_PERSISTENCE_SYNC_AOF_START, similar to REDISMODULE_SUBEVENT_PERSISTENCE_AOF_START which is async. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 12 Jan, 2022 1 commit
-
-
Binbin authored
Use `getFullCommandName` to get the full name of the command. It can also get the full name of the subcommand, like "script|help". Before: ``` > SCRIPT HELP (error) NOPERM this user has no permissions to run the 'help' command or its subcommand > ACL LOG 7) "object" 8) "help" ``` After: ``` > SCRIPT HELP (error) NOPERM this user has no permissions to run the 'script|help' command > ACL LOG 7) "object" 8) "script|help" ``` Fix #10094
-
- 11 Jan, 2022 7 commits
-
-
Binbin authored
This commit adds some tests that the test cases will access the keys with expiration time set in the script call. There was no test case for this part before. See #10080 Also there is a test will cover #1525. we block the time so that the key can not expire in the middle of the script execution. Other changes: 1. Delete `evalTimeSnapshot` and just use `scriptTimeSnapshot` in it's place. 2. Some cleanups to scripting.tcl. 3. better names for tests that run in a loop to make them distinctable
-
Ozan Tezcan authored
Added a pool for temporary client objects to reuse in module operations. By reusing temporary clients, we are avoiding expensive createClient()/freeClient() calls and improving performance of RM_BlockClient() and RM_GetThreadSafeContext() calls. This commit contains two optimizations: 1 - RM_BlockClient() and RM_GetThreadSafeContext() calls create temporary clients and they are freed in RM_UnblockClient() and RM_FreeThreadSafeContext() calls respectively. Creating/destroying client object takes quite time. To avoid that, added a pool of temporary clients. Pool expands when more clients are needed. Also, added a cron function to shrink the pool and free unused clients after some time. Pool starts with zero clients in it. It does not have max size and can grow unbounded as we need it. We will keep minimum of 8 temporary clients in the pool once created. Keeping small amount of clients to avoid client allocation costs if temporary clients are required after some idle period. 2 - After unblocking a client (RM_UnblockClient()), one byte is written to pipe to wake up Redis main thread. If there are many clients that will be unblocked, each operation requires one write() call which is quite expensive. Changed code to avoid subsequent calls if possible. There are a few more places that need temporary client objects (e.g RM_Call()). These are now using the same temporary client pool to make things more centralized.
-
Oran Agra authored
Syntax: `COMMAND DOCS [<command name> ...]` Background: Apparently old version of hiredis (and thus also redis-cli) can't support more than 7 levels of multi-bulk nesting. The solution is to move all the doc related metadata from COMMAND to a new COMMAND DOCS sub-command. The new DOCS sub-command returns a map of commands (not an array like in COMMAND), And the same goes for the `subcommands` field inside it (also contains a map) Besides that, the remaining new fields of COMMAND (hints, key-specs, and sub-commands), are placed in the outer array rather than a nested map. this was done mainly for consistency with the old format. Other changes: --- * Allow COMMAND INFO with no arguments, which returns all commands, so that we can some day deprecated the plain COMMAND (no args) * Reduce the amount of deferred replies from both COMMAND and COMMAND DOCS, especially in the inner loops, since these create many small reply objects, which lead to many small write syscalls and many small TCP packets. To make this easier, when populating the command table, we count the history, args, and hints so we later know their size in advance. Additionally, the movablekeys flag was moved into the flags register. * Update generate-commands-json.py to take the data from both command, it now executes redis-cli directly, instead of taking input from stdin. * Sub-commands in both COMMAND (and COMMAND INFO), and also COMMAND DOCS, show their full name. i.e. CONFIG * GET will be shown as `config|get` rather than just `get`. This will be visible both when asking for `COMMAND INFO config` and COMMAND INFO config|get`, but is especially important for the later. i.e. imagine someone doing `COMMAND INFO slowlog|get config|get` not being able to distinguish between the two items in the array response.
-
guybe7 authored
We must fail RM_SubscribeToServerEvent in case a module, that was compiled with a new redismodule.h, tries to subscribe to an event that doesn't exist on an old redis-server
-
Binbin authored
It used to return `$-1` in RESP2, now we will return `*-1`. This is a bug in redis 6.2 when COUNT was added, the `COUNT` option was introduced in #8179. Fix #10089. the documentation of [LPOP](https://redis.io/commands/lpop) says ``` When called without the count argument: Bulk string reply: the value of the first element, or nil when key does not exist. When called with the count argument: Array reply: list of popped elements, or nil when key does not exist. ```
-
小令童鞋 authored
This is a recent regression from the Redis Functions commits
-
Madelyn Olson authored
-
- 10 Jan, 2022 4 commits
-
-
chenyang8094 authored
1. Ban whitespace characters in `appenddirname` 2. Handle the case where `appendfilename` contains spaces (for backwards compatibility)
-
Binbin authored
The following steps will crash redis-server: ``` [root]# cat crash PSYNC replicationid -1 SLOWLOG GET GET key [root]# nc 127.0.0.1 6379 < crash ``` This one following #10020 and the crash was reported in #10076. Other changes about the output info: 1. Cmd with a full name by using `getFullCommandName`, now it will print the right subcommand name like `slowlog|get`. 2. Print the full client info by using `catClientInfoString`, the info is also valuable.:
-
Wen Hui authored
readonly/readwrite only set client flags for slave in cluster mode, so it should be ok for setting ok-stale and ok-loading command flag
-
Madelyn Olson authored
Changed latency percentile output to omit trailing 0s and periods
-
- 09 Jan, 2022 3 commits
-
-
Binbin authored
The following error commands will crash redis-server: ``` > get| Error: Server closed the connection > get|set Error: Server closed the connection > get|other ``` The reason is in #9504, we use `lookupCommandBySds` for find the container command. And it split the command (argv[0]) with `|`. If we input something like `get|other`, after the split, `get` will become a valid command name, pass the `ERR unknown command` check, and finally crash in `addReplySubcommandSyntaxError` In this case we do not need to split the command name with `|` and just look in the commands dict to find if `argv[0]` is a container command. So this commit introduce a new function call `isContainerCommandBySds` that it will return true if a command name is a container command. Also with the old code, there is a incorrect error message: ``` > config|get set (error) ERR Unknown subcommand or wrong number of arguments for 'set'. Try CONFIG|GET HELP. ``` The crash was reported in #10070.
-
Itamar Haber authored
-
YEONCHEOL JANG authored
Fixes typo for redis.conf
-
- 07 Jan, 2022 1 commit
-
-
guybe7 authored
Fix #9410 Crucial for the ms and sequence deltas, but I changed all calls, just in case (e.g. "flags") Before this commit: `ms_delta` and `seq_delta` could have overflown, causing `currid` to be wrong, which in turn would cause `streamTrim` to trim the entire rax node (see new test)
-
- 06 Jan, 2022 3 commits
-
-
Viktor Söderqvist authored
With this rule, the script to generate commands.c from JSON runs whenever commands.o is built if any of commands/*.json are modified. Without such rule, it's easy to forget to run the script when updating the JSON files. It's a follow-up on #9656 and #9951.
-
Meir Shpilraien (Spielrein) authored
# Redis Function Libraries This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906. Libraries purpose is to provide a better code sharing between functions by allowing to create multiple functions in a single command. Functions that were created together can safely share code between each other without worrying about compatibility issues and versioning. Creating a new library is done using 'FUNCTION LOAD' command (full API is described below) This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library: * name - name of the library * engine - engine used to create the library * code - library code * description - library description * functions - the functions exposed by the library When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo. Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo. As a result, the engine will create one or more functions by calling 'libraryCreateFunction'. The new funcion will be added to the newly created libraryInfo. So far Everything is happening locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply freeing the libraryInfo. After the library info is fully constructed we start the joining phase by which we will join the new library to the other libraries currently exist on Redis. The joining phase make sure there is no function collision and add the library to the librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact same way as functionCtx was used (with respect to RDB loading, replicatio, ...). The only difference is that apart from function dictionary (maps function name to functionInfo object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object. ## New API ### FUNCTION LOAD `FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>` Create a new library with the given parameters: * ENGINE - REPLACE Engine name to use to create the library. * LIBRARY NAME - The new library name. * REPLACE - If the library already exists, replace it. * DESCRIPTION - Library description. * CODE - Library code. Return "OK" on success, or error on the following cases: * Library name already taken and REPLACE was not used * Name collision with another existing library (even if replace was uses) * Library registration failed by the engine (usually compilation error) ## Changed API ### FUNCTION LIST `FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]` Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries. ### INFO MEMORY Added number of libraries to `INFO MEMORY` ### Commands flags `DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands as commands that add new data to the dateset (functions are data) and so we want to disallows to run those commands on OOM. ## Removed API * FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906 * FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899 ## Lua engine changes When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call this run the loading run. Loading run is not a usual script run, it is not possible to invoke any Redis command from within the load run. Instead there is a new API provided by `library` object. The new API's: * `redis.log` - behave the same as `redis.log` * `redis.register_function` - register a new function to the library The loading run purpose is to register functions using the new `redis.register_function` API. Any attempt to use any other API will result in an error. In addition, the load run is has a time limit of 500ms, error is raise on timeout and the entire operation is aborted. ### `redis.register_function` `redis.register_function(<function_name>, <callback>, [<description>])` This new API allows users to register a new function that will be linked to the newly created library. This API can only be called during the load run (see definition above). Any attempt to use it outside of the load run will result in an error. The parameters pass to the API are: * function_name - Function name (must be a Lua string) * callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro * description - Function description, optional (must be a Lua string). ### Example The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively: ``` local function f1(keys, args) return 1 end local function f2(keys, args) return 2 end redis.register_function('f1', f1) redis.register_function('f2', f2) ``` Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the functions and not as global. ### Technical Details On the load run we only want the user to be able to call a white list on API's. This way, in the future, if new API's will be added, the new API's will not be available to the load run unless specifically added to this white list. We put the while list on the `library` object and make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set the `globals` of a function (and all the function it creates). Before starting the load run we create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure to set global protection on this table just like the general global protection already exists today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) to set `g` as the global table of the load run. After the load run finished we update `g` metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals), we also pop out the `library` object as we do not need it anymore. This way, any function that was created on the load run (and will be invoke using `fcall`) will see the default globals as it expected to see them and will not have the `library` API anymore. An important outcome of this new approach is that now we can achieve a distinct global table for each library (it is not yet like that but it is very easy to achieve it now). In the future we can decide to remove global protection because global on different libraries will not collide or we can chose to give different API to different libraries base on some configuration or input. Notice that this technique was meant to prevent errors and was not meant to prevent malicious user from exploit it. For example, the load run can still save the `library` object on some local variable and then using in `fcall` context. To prevent such a malicious use, the C code also make sure it is running in the right context and if not raise an error.
-
Ozan Tezcan authored
Callers of redisFork() are logging `strerror(errno)` on failure. `errno` is not set when there is already a child process, causing printing current value of errno which was set before `redisFork()` call. Setting errno to EEXIST on this failure to provide more meaningful error message.
-
- 05 Jan, 2022 4 commits
-
-
filipe oliveira authored
Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462) # Short description The Redis extended latency stats track per command latencies and enables: - exporting the per-command percentile distribution via the `INFO LATENCYSTATS` command. **( percentile distribution is not mergeable between cluster nodes ).** - exporting the per-command cumulative latency distributions via the `LATENCY HISTOGRAM` command. Using the cumulative distribution of latencies we can merge several stats from different cluster nodes to calculate aggregate metrics . By default, the extended latency monitoring is enabled since the overhead of keeping track of the command latency is very small. If you don't want to track extended latency metrics, you can easily disable it at runtime using the command: - `CONFIG SET latency-tracking no` By default, the exported latency percentiles are the p50, p99, and p999. You can alter them at runtime using the command: - `CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"` ## Some details: - The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command was called for the first time. - With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable vs this branch. - We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf ) ## `INFO LATENCYSTATS` exposition format - Format: `latency_percentiles_usec_<CMDNAME>:p0=XX,p50....` ## `LATENCY HISTOGRAM [command ...]` exposition format Return a cumulative distribution of latencies in the format of a histogram for the specified command names. The histogram is composed of a map of time buckets: - Each representing a latency range, between 1 nanosecond and roughly 1 second. - Each bucket covers twice the previous bucket's range. - Empty buckets are not printed. - Everything above 1 sec is considered +Inf. - At max there will be log2(1000000000)=30 buckets We reply a map for each command in the format: `<command name> : { `calls`: <total command calls> , `histogram` : { <bucket 1> : latency , < bucket 2> : latency, ... } }` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
sundb authored
Following #10038. This PR introduces two changes. 1. Show the elapsed time of a single test in the test output, in order to have a more detailed understanding of the changes in test run time. 2. Speedup two tests related to `key-load-delay` configuration. other tests do not seem to be affected by #10003.
-
Binbin authored
-
Ozan Tezcan authored
-
- 04 Jan, 2022 8 commits
-
-
Binbin authored
The purpose of this commit is to add some tests to cover #5299, which was fixed in #5300 but without tests. This commit should close #5306 and #5299.
-
Yuta Hongo authored
About RESP3 an ordered collection of key-value pairs, keys and value can be any other RESP3 type, but a key should be string in JSON spec.
-
yoav-steinberg authored
This makes redis-cli --replica much faster and reduces COW/fork risks on server side. This commit also improves the RDB filtering via REPLCONF rdb-filter-only to support no "include" specifiers at all.
-
Matthieu MOREL authored
This sets up dependabot to check weekly updates for pip and github-actions dependencies. If it finds an update it will create a PR to update the dependency. More information can be found here It includes the update of: * vmactions/freebsd-vm from 0.1.4 to 0.1.5 * codespell from 2.0.0 to 2.1.0 Also includes spelling fixes found by the latest version of codespell. Includes a dedicated .codespell folder so dependabot can read a requirements.txt file and every files dedicated to codespell can be grouped in the same place Co-Authored-By:
Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-Authored-By:
MOREL Matthieu <matthieu.morel@cnp.fr>
-
Binbin authored
In monitor/pubsub mode, if the server closes the connection, for example, use `CLIENT KILL`, redis-cli will exit directly without printing any error messages. This commit ensures that redis-cli will try to print the error messages before exiting. Also there is a minor cleanup for restart, see the example below. before: ``` 127.0.0.1:6379> monitor OK [root@ redis]# 127.0.0.1:6379> subscribe channel Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "channel" 3) (integer) 1 [root@ redis]# 127.0.0.1:6379> restart 127.0.0.1:6379> get keyUse 'restart' only in Lua debugging mode. (nil) ``` after: ``` 127.0.0.1:6379> monitor OK Error: Server closed the connection [root@ redis]# 127.0.0.1:6379> subscribe channel Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "channel" 3) (integer) 1 Error: Server closed the connection [root@ redis]# 127.0.0.1:6379> restart Use 'restart' only in Lua debugging mode. ```
-
王辉 authored
Older version of GNU Make (<4.3) required quoting of number signs (#) to avoid them being treated as a comment. Newer versions will treat this quote as a literal. This issue and a proposed solution is discussed here: https://lists.gnu.org/archive/html/info-gnu/2020-01/msg00004.html Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
guybe7 authored
Creating fork (or even a foreground SAVE) during a transaction breaks the atomicity of the transaction. In addition to that, it could mess up the propagated transaction to the AOF file. This change blocks SAVE, PSYNC, SYNC and SHUTDOWN from being executed inside MULTI-EXEC. It does that by adding a command flag, so that modules can flag their commands with that flag too. Besides it changes BGSAVE, BGREWRITEAOF, and CONFIG SET appendonly, to turn the scheduled flag instead of forking righ taway. Other changes: * expose `protected`, `no-async-loading`, and `no_multi` flags in COMMAND command * add a test to validate propagation of FLUSHALL inside a transaction. * add a test to validate how CONFIG SET that errors reacts in a transaction Co-authored-by:
Oran Agra <oran@redislabs.com>
-
zhaozhao.zz authored
This would mean that the effects of `CONFIG SET maxmemory` may not be visible once the command returns. That could anyway happen since incremental eviction was added in redis 6.2 (see #7653) We do this to fix one of the propagation bugs about eviction see #9890 and #10014.
-
- 03 Jan, 2022 4 commits
-
-
chenyang8094 authored
Implement Multi-Part AOF mechanism to avoid overheads during AOFRW. Introducing a folder with multiple AOF files tracked by a manifest file. The main issues with the the original AOFRW mechanism are: * buffering of commands that are processed during rewrite (consuming a lot of RAM) * freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it. * double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files) The main modifications of this PR: 1. Remove the AOF rewrite buffer and related code. 2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type, it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the incremental commands since the last AOFRW. 3. Use a AOF manifest file to record and manage these AOF files mentioned above. 4. The original configuration of `appendfilename` will be the base part of the new file name, for example: `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof` 5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename` 6. Remove the `aof_rewrite_buffer_length` field in info. 7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs. It also gives users the opportunity to preserve the history AOFs. just for testing use now. 8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now), we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately. 9. Support upgrade (load) data from old version redis. 10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and manifest file will be placed in this directory. 11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if `aof-load-truncated` is enabled. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Meir Shpilraien (Spielrein) authored
OOM Error did not raise on functions due to a bug. Added test to verify the fix.
-
Madelyn Olson authored
Implement the ability for cluster nodes to advertise their location with extension messages.
-
Harkrishn Patro authored
This commit implements a sharded pubsub implementation based off of shard channels. Co-authored-by:
Harkrishn Patro <harkrisp@amazon.com> Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 02 Jan, 2022 1 commit
-
-
Binbin authored
This commit adds DUMP RESTORES tests for the -x and -X options. I wanted to add it in #9980 which introduce the -X option, but back then i failed due to some errors (related to redis-cli call).
-