- 27 Jan, 2022 1 commit
-
-
Yossi Gottlieb authored
There are some inevitable changes between the old and new output, as a result of the different `commands.json` semantics.
-
- 26 Jan, 2022 6 commits
-
-
Binbin authored
SET is a R+W command, because it can also do `GET` on the data. SET without GET is a write-only command. SET with GET is a read+write command. In #9974, we added ACL to let users define write-only access. So when the user uses SET with GET option, and the user doesn't have the READ permission on the key, we need to reject it, but we rather not reject users with write-only permissions from using the SET command when they don't use GET. In this commit, we add a `getkeys_proc` function to control key flags in SET command. We also add a new key spec flag (VARIABLE_FLAGS) means that some keys might have different flags depending on arguments. We also handle BITFIELD command, add a `bitfieldGetKeys` function. BITFIELD GET is a READ ONLY command. BITFIELD SET or BITFIELD INCR are READ WRITE commands. Other changes: 1. SET GET was added in 6.2, add the missing since in set.json 2. Added tests to cover the changes in acl-v2.tcl 3. Fix some typos in server.h and cleanups in acl-v2.tcl Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Oran Agra authored
This PR attempts to solve two problems that happen sometime in valgrind: `ERR Background save already in progress` and `not bgsave not aborted` the test used to populate the database with DEBUG, which didn't increment the dirty counter, so couldn't trigger an automatic bgsave. then it used a manual bgsave, and aborted it (when it got aborted it populated the dirty counter), and then it tried to do another bgsave. that other bgsave could have failed if the automatic one already started.
-
Oran Agra authored
Failed on a non-valgrind run. on this line: ``` assert_equal 0 [$slave exists k] ``` the condition in `keyIsExpired` is `now > when`. so if the test is really fast, maybe it can get to EXISTS exactly 1000 milliseconds after the expiration was set, and the key isn't yet gone)
-
guybe7 authored
-
Binbin authored
Forgot to handle it in #10135.
-
Madelyn Olson authored
This is an attempt to fix some of the issues with the cluster mode tests we are seeing in the daily run. The test is trying to incrementally adds a bunch of publish messages, expecting that eventually one of them will overflow. The tests stops one of the processes, so it expects that just that one Redis node will overflow. I think the test is flaky because under certain circumstances multiple links are getting disconnected, not just the one that is stalled.
-
- 25 Jan, 2022 6 commits
-
-
Viktor Söderqvist authored
Before: ``` 127.0.0.1:6379> command info get 1) 1) "get" 2) (integer) 2 3) 1~ readonly 2~ fast 4) (integer) 1 5) (integer) 1 6) (integer) 1 7) 1~ @read 2~ @string 3~ @fast 8) (empty set) 9) 1~ 1# "flags" => 1~ RO 2~ access 2# "begin_search" => 1# "type" => "index" 2# "spec" => 1# "index" => (integer) 1 3# "find_keys" => 1# "type" => "range" 2# "spec" => 1# "lastkey" => (integer) 0 2# "keystep" => (integer) 1 3# "limit" => (integer) 0 10) (empty set) ``` After: ``` 127.0.0.1:6379> command info get 1) 1) "get" 2) (integer) 2 3) 1~ readonly 2~ fast 4) (integer) 1 5) (integer) 1 6) (integer) 1 7) 1~ @read 2~ @string 3~ @fast 8) (empty set) 9) 1~ 1# "flags" => 1~ RO 2~ access 2# "begin_search" => 1# "type" => "index" 2# "spec" => 1# "index" => (integer) 1 3# "find_keys" => 1# "type" => "range" 2# "spec" => 1# "lastkey" => (integer) 0 2# "keystep" => (integer) 1 3# "limit" => (integer) 0 10) (empty set) ```
-
Meir Shpilraien (Spielrein) authored
Added the following statistics (per engine) to FUNCTION STATS command: * number of functions * number of libraries Output example: ``` > FUNCTION stats 1) "running_script" 2) (nil) 3) "engines" 4) 1) "LUA" 2) 1) "libraries_count" 2) (integer) 1 3) "functions_count" 4) (integer) 1 ``` To collect the stats, added a new dictionary to libraries_ctx that contains for each engine, the engine statistics representing the current libraries_ctx. Update the stats on: 1. Link library to libraries_ctx 2. Unlink library from libraries_ctx 3. Flushing libraries_ctx
-
yoav-steinberg authored
Set commands.c's merge driver to binary so when it conflicts during a merge git will leave the local version unmodified. This way our Makefile will rebuild it based on src/commands/*.json before trying to compile it. Otherwise the file gets modified with merge conflict markers and gets the same timestamp as the *.json files, so the Makefile doesn't attempt to rebuild it before compiling and we get a compilation error.
-
chenyang8094 authored
-
Madelyn Olson authored
This PR aims to improve the flags associated with some commands and adds various tests around these cases. Specifically, it's concerned with commands which declare keys but have no ACL flags (think `EXISTS`), the user needs either read or write permission to access this type of key. This change is primarily concerned around commands in three categories: # General keyspace commands These commands are agnostic to the underlying data outside of side channel attacks, so they are not marked as ACCESS. * TOUCH * EXISTS * TYPE * OBJECT 'all subcommands' Note that TOUCH is not a write command, it could be a side effect of either a read or a write command. # Length and cardinality commands These commands are marked as NOT marked as ACCESS since they don't return actual user strings, just metadata. * LLEN * STRLEN * SCARD * HSTRLEN # Container has member commands These commands return information about the existence or metadata about the key. These commands are NOT marked as ACCESS since the check of membership is used widely in write commands e.g. the response of HSET. * SISMEMBER * HEXISTS # Intersection cardinality commands These commands are marked as ACCESS since they process data to compute the result. * PFCOUNT * ZCOUNT * ZINTERCARD * SINTERCARD
-
Madelyn Olson authored
Correctly handle variable arity checks in scripts
-
- 24 Jan, 2022 7 commits
-
-
Oran Agra authored
The edit i made to #10098 before merging was broken. (INFO is forbidden in LOADING)
-
chenyang8094 authored
* Refactor EVAL timeout test * since the test used r config set appendonly yes which generates a rewrite, it missed it's purpose * Fix the bug that start_server returns before redis starts ready, which affects when multiple tests share the same dir. * Elapsed time tracking no loner needed Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Binbin authored
For some complex data types, server.dirty actually counts the number of elements that have been changed. And in FLUSHDB or FLUSHALL, we count the number of keys. So the word "key" is not strictly correct and is outdated. Some discussion can be seen at #8140.
-
yoav-steinberg authored
In #10025 we added a mechanism for flagging certain properties for Redis Functions. This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. One good reason for this, other than the added functionality is because it addresses the poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details. Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all. This PR: - Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025. - Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for additional script annotations and even engine selection (instead of just lua) for scripts. - Provides backwards compatibility so scripts without the new annotations will behave as they did before. - Appropriate tests. - Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will return the `MASTERDOWN` error as before. - Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that won't work unless the script is explicitly marked as `allow-stale`. Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the first line with a single line feed character.
-
Viktor Söderqvist authored
The keyspec API is not yet released and there is a plan to change it in #10108, which is going to be included in RC2. Therefore, we hide it in RC1 to avoid introducing a breaking change in RC2. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
chenyang8094 authored
-
ny0312 authored
* Fix flaky cluster test "Disconnect link when send buffer limit reached" * Fix flaky cluster test "Each node has two links with each peer" Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 23 Jan, 2022 2 commits
-
-
Binbin authored
A test failure was reported in Daily CI (test-centos7-tls). `CKQUORUM detects failover authorization cannot be reached`. ``` CKQUORUM detects failover authorization cannot be reached: FAILED: Expected 'invalid command name "OK 4 usable Sentinels. Quorum and failover authorization can be reached"' to match '*NOQUORUM*' ``` It seems that current sentinel does not confirm that the other sentinels are actually `down`, and then check the quorum. It at least take 3 seconds on my machine, and we can see there will be a timing issue with the hard code `after 5000`. In this commit, we check the response of `SENTINEL SENTINELS mymaster` to ensure that other sentinels are actually `down` in the view the current sentinel. Solve the timing issue due to sentinel monitor mechanism.
-
Binbin authored
Summary of changes: 1. Rename `redisCommand->name` to `redisCommand->declared_name`, it is a const char * for native commands and SDS for module commands. 2. Store the [sub]command fullname in `redisCommand->fullname` (sds). 3. List subcommands in `ACL CAT` 4. List subcommands in `COMMAND LIST` 5. `moduleUnregisterCommands` now will also free the module subcommands. 6. RM_GetCurrentCommandName returns full command name Other changes: 1. Add `addReplyErrorArity` and `addReplyErrorExpireTime` 2. Remove `getFullCommandName` function that now is useless. 3. Some cleanups about `fullname` since now it is SDS. 4. Delete `populateSingleCommand` function from server.h that is useless. 5. Added tests to cover this change. 6. Add some module unload tests and fix the leaks 7. Make error messages uniform, make sure they always contain the full command name and that it's quoted. 7. Fixes some typos see the history in #9504, fixes #10124 Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
guybe7 <guy.benoish@redislabs.com>
-
- 22 Jan, 2022 1 commit
-
-
guybe7 authored
Recently we added extensive support for sub-commands in for redis 7.0, this meant that the old ACL mechanism for sub-commands wasn't needed, or actually was improved (to handle both include and exclude control, like for commands), but only for real sub-commands. The old mechanism in ACL was renamed to first-arg, and was able to match the first argument of any command (including sub-commands). We now realized that we might wanna completely delete that first-arg feature some day, so the first step was not to give it new capabilities in 7.0 and it didn't have before. Changes: 1. ACL: Block the first-arg mechanism on subcommands (we keep if in non-subcommands for backward compatibility) 2. COMMAND: When looking up a command, insist the command name doesn't contain extra words. Example: When a user issues `GET key` we want `lookupCommand` to return `getCommand` but when if COMMAND calls `lookupCommand` with `get|key` we want it to fail. Other changes: 1. ACLSetUser: prevent a redundant command lookup
-
- 20 Jan, 2022 5 commits
-
-
Madelyn Olson authored
* Implemented selectors which provide multiple different sets of permissions to users * Implemented key based permissions * Added a new ACL dry-run command to test permissions before execution * Updated module APIs to support checking key based permissions Co-authored-by:
Oran Agra <oran@redislabs.com>
-
guybe7 authored
Adding command tips (see https://redis.io/topics/command-tips) to commands. Breaking changes: 1. Removed the "random" and "sort_for_script" flags. They are now command tips. (this isn't affecting redis behavior since #9812, but could affect some client applications that's relying on COMMAND command flags) Summary of changes: 1. add BLOCKING flag (new flag) for all commands that could block. The ACL category with the same name is now implicit. 2. move RANDOM flag to a `nondeterministic_output` tip 3. move SORT_FOR_SCRIPT flag to `nondeterministic_output_order` tip 3. add REQUEST_POLICY and RESPONSE_POLICY where appropriate as documented in the tips 4. deprecate (ignore) the `random` flag for RM_CreateCommand Other notes: 1. Proxies need to send `RANDOMKEY` to all shards and then select one key randomly. The other option is to pick a random shard and transfer `RANDOMKEY `to it, but that scheme fails if this specific shard is empty 2. Remove CMD_RANDOM from `XACK` (i.e. XACK does not have RANDOM_OUTPUT) It was added in 9e4fb96c, I guess by mistake. Also from `(P)EXPIRETIME` (new command, was flagged "random" by mistake). 3. Add `nondeterministic_output` to `OBJECT ENCODING` (for the same reason `XTRIM` has it: the reply may differ depending on the internal representation in memory) 4. RANDOM on `HGETALL` was wrong (there due to a limitation of the old script sorting logic), now it's `nondeterministic_output_order` 5. Unrelated: Hide CMD_PROTECTED from COMMAND
-
Meir Shpilraien (Spielrein) authored
The PR added the missing verification for functions on redis-check-rdb. The verification only verifies the rdb structure and does not try to load the functions code and verify more advance checks (like compilation of the function code).
-
perryitay authored
Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963) Some modules might perform a long-running logic in different stages of Redis lifetime, for example: * command execution * RDB loading * thread safe context During this long-running logic Redis is not responsive. This PR offers 1. An API to process events while a busy command is running (`RM_Yield`) 2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy jobs which can also be used by modules (`allow-busy`) 3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only after `busy-reply-threshold` 4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`), but either way, the processing is throttled to the server hz rate. 5. Allow modules to Yield to redis background tasks, but not to client commands * rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`) Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Madelyn Olson authored
* Fix double key declaration for renamenx * Change the flag from UPDATE to INSERT, since it can not affect the data in the key
-
- 19 Jan, 2022 6 commits
-
-
Meir Shpilraien (Spielrein) authored
Function PR was merged without AOF rw support because we thought this feature was going to be removed on Redis 7. Tests was added on aofrw.tcl Other existing aofrw tests where slow due to unwanted rdb-key-save-delay Co-authored-by:
Oran Agra <oran@redislabs.com>
-
David CARLIER authored
Add support for getting the RSS in OpenBSD
-
Binbin authored
In #10122, we modify the key spec flags to `RO` and `ACCESS`. But forgot to call generate-command-code.py. Also formatted it to follow the Python PEP8.
-
Wen Hui authored
-
Ozan Tezcan authored
Tests were not using loop index as node id, checking replica count of the same node over and over.
-
Ozan Tezcan authored
was eating too much memory, and taking too long with valgrind
-
- 18 Jan, 2022 5 commits
-
-
Oran Agra authored
The new ACL key based permissions in #9974 require the key-specs (#8324) to have more explicit flags rather than just READ and WRITE. See discussion in #10040 This PR defines two groups of flags: One about how redis internally handles the key (mutually-exclusive). The other is about the logical operation done from the user's point of view (3 mutually exclusive write flags, and one read flag, all optional). In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or insert-only, we flag it as `RW` or `UPDATE`. here's the definition from the code: ``` /* Key-spec flags * * -------------- */ /* The following refer what the command actually does with the value or metadata * of the key, and not necessarily the user data or how it affects it. * Each key-spec may must have exaclty one of these. Any operation that's not * distinctly deletion, overwrite or read-only would be marked as RW. */ #define CMD_KEY_RO (1ULL<<0) /* Read-Only - Reads the value of the key, but * doesn't necessarily returns it. */ #define CMD_KEY_RW (1ULL<<1) /* Read-Write - Modifies the data stored in the * value of the key or its metadata. */ #define CMD_KEY_OW (1ULL<<2) /* Overwrite - Overwrites the data stored in * the value of the key. */ #define CMD_KEY_RM (1ULL<<3) /* Deletes the key. */ /* The follwing refer to user data inside the value of the key, not the metadata * like LRU, type, cardinality. It refers to the logical operation on the user's * data (actual input strings / TTL), being used / returned / copied / changed, * It doesn't refer to modification or returning of metadata (like type, count, * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE. * Each key-spec may have one of the writes with or without access, or none: */ #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from * the value of the key. */ #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may * depend on the old value. */ #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of, * modification or deletion of existing data. */ #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content * from the value of the key. */ ``` Unrelated changes: - generate-command-code.py is only compatible with python3 (modified the shabang) - generate-command-code.py print file on json parsing error - rename `shard_channel` key-spec flag to just `channel`. - add INCOMPLETE flag in input spec of SORT and SORT_RO
-
Wang Yuan authored
When I used C++ to develop a redis module. i used `string.data()` as the second parameter `ele` of `RedisModule_DigestAddStringBuffer`, but there is a warning, since we never change the `ele`, i think we should use `const char` for it. This PR adds const to just a handful of module APIs that required it, all not very widely used. The implication is a breaking change in terms of compilation error that's easy to resolve, and no ABI impact. The affected APIs are around Digest, Info injection, and Cluster bus messages.
-
Ozan Tezcan authored
Modules can now register sockets/pipe to the Redis main thread event loop and do network operations asynchronously. Previously, modules had to maintain an event loop and another thread for asynchronous network operations. Also, if a module is calling API functions after doing some network operations, it had to synchronize its event loop thread's access with Redis main thread by locking the GIL, causing contention on the lock. After this commit, no synchronization is needed as module can operate in Redis main thread context. So, this commit may improve the performance for some use cases. Added three functions to the module API: * RedisModule_EventLoopAdd(int fd, int mask, RedisModuleEventLoopFunc func, void *user_data) * RedisModule_EventLoopDel(int fd, int mask) * RedisModule_EventLoopAddOneShot(RedisModuleEventLoopOneShotFunc func, void *user_data) - This function can be called from other threads to trigger callback on Redis main thread. Callback will be triggered only once. If Redis main thread is sleeping, this call will wake up the Redis main thread. Event loop callbacks are called by Redis main thread after locking the GIL. Inside callbacks, modules can operate as if they are holding the GIL. Added REDISMODULE_EVENT_EVENTLOOP event with two subevents: * REDISMODULE_SUBEVENT_EVENTLOOP_BEFORE_SLEEP * REDISMODULE_SUBEVENT_EVENTLOOP_AFTER_SLEEP These events are for modules that want to participate in the before and after sleep action. e.g It might be useful to implement batching : Read data from the network, write all to a file in one go on BEFORE_SLEEP event.
-
Yossi Gottlieb authored
This extends the previous fix (#10049) to address any form of non-printable or whitespace character (including newlines, quotes, non-printables, etc.) Also, removes the limitation on appenddirname, to align with the way filenames are handled elsewhere in Redis.
-
Meir Shpilraien (Spielrein) authored
Following discussion on: https://github.com/redis/redis/issues/9899#issuecomment-1014689385 Raise error if unknows parameter is given to `FUNCTION LOAD`. Before the fix: ``` 127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)" OK ``` After the fix: ``` 127.0.0.1:6379> function load LUA lib2 foo bar "local function test1() return 5 end redis.register_function('test1', test1)" (error) ERR Unkowns option given: foo ```
-
- 17 Jan, 2022 1 commit
-
-
Oran Agra authored
1. enable diskless replication by default 2. add a new config named repl-diskless-sync-max-replicas that enables replication to start before the full repl-diskless-sync-delay was reached. 3. put replica online sooner on the master (see below) 4. test suite uses repl-diskless-sync-delay of 0 to be faster 5. a few tests that use multiple replica on a pre-populated master, are now using the new repl-diskless-sync-max-replicas 6. fix possible timing issues in a few cluster tests (see below) put replica online sooner on the master ---------------------------------------------------- there were two tests that failed because they needed for the master to realize that the replica is online, but the test code was actually only waiting for the replica to realize it's online, and in diskless it could have been before the master realized it. changes include two things: 1. the tests wait on the right thing 2. issues in the master, putting the replica online in two steps. the master used to put the replica as online in 2 steps. the first step was to mark it as online, and the second step was to enable the write event (only after getting ACK), but in fact the first step didn't contains some of the tasks to put it online (like updating good slave count, and sending the module event). this meant that if a test was waiting to see that the replica is online form the point of view of the master, and then confirm that the module got an event, or that the master has enough good replicas, it could fail due to timing issues. so now the full effect of putting the replica online, happens at once, and only the part about enabling the writes is delayed till the ACK. fix cluster tests -------------------- I added some code to wait for the replica to sync and avoid race conditions. later realized the sentinel and cluster tests where using the original 5 seconds delay, so changed it to 0. this means the other changes are probably not needed, but i suppose they're still better (avoid race conditions)
-