- 20 Jan, 2022 1 commit
-
-
perryitay authored
Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963) Some modules might perform a long-running logic in different stages of Redis lifetime, for example: * command execution * RDB loading * thread safe context During this long-running logic Redis is not responsive. This PR offers 1. An API to process events while a busy command is running (`RM_Yield`) 2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy jobs which can also be used by modules (`allow-busy`) 3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only after `busy-reply-threshold` 4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`), but either way, the processing is throttled to the server hz rate. 5. Allow modules to Yield to redis background tasks, but not to client commands * rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`) Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 11 Jan, 2022 1 commit
-
-
Binbin authored
This commit adds some tests that the test cases will access the keys with expiration time set in the script call. There was no test case for this part before. See #10080 Also there is a test will cover #1525. we block the time so that the key can not expire in the middle of the script execution. Other changes: 1. Delete `evalTimeSnapshot` and just use `scriptTimeSnapshot` in it's place. 2. Some cleanups to scripting.tcl. 3. better names for tests that run in a loop to make them distinctable
-
- 21 Dec, 2021 1 commit
-
-
zhugezy authored
# Background The main goal of this PR is to remove relevant logics on Lua script verbatim replication, only keeping effects replication logic, which has been set as default since Redis 5.0. As a result, Lua in Redis 7.0 would be acting the same as Redis 6.0 with default configuration from users' point of view. There are lots of reasons to remove verbatim replication. Antirez has listed some of the benefits in Issue #5292: >1. No longer need to explain to users side effects into scripts. They can do whatever they want. >2. No need for a cache about scripts that we sent or not to the slaves. >3. No need to sort the output of certain commands inside scripts (SMEMBERS and others): this both simplifies and gains speed. >4. No need to store scripts inside the RDB file in order to startup correctly. >5. No problems about evicting keys during the script execution. When looking back at Redis 5.0, antirez and core team decided to set the config `lua-replicate-commands yes` by default instead of removing verbatim replication directly, in case some bad situations happened. 3 years later now before Redis 7.0, it's time to remove it formally. # Changes - configuration for lua-replicate-commands removed - created config file stub for backward compatibility - Replication script cache removed - this is useless under script effects replication - relevant statistics also removed - script persistence in RDB files is also removed - Propagation of SCRIPT LOAD and SCRIPT FLUSH to replica / AOF removed - Deterministic execution logic in scripts removed (i.e. don't run write commands after random ones, and sorting output of commands with random order) - the flags indicating which commands have non-deterministic results are kept as hints to clients. - `redis.replicate_commands()` & `redis.set_repl()` changed - now `redis.replicate_commands()` does nothing and return an 1 - ...and then `redis.set_repl()` can be issued before `redis.replicate_commands()` now - Relevant TCL cases adjusted - DEBUG lua-always-replicate-commands removed # Other changes - Fix a recent bug comparing CLIENT_ID_AOF to original_client->flags instead of id. (introduced in #9780) Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 02 Dec, 2021 1 commit
-
-
meir@redislabs.com authored
Redis function unit is located inside functions.c and contains Redis Function implementation: 1. FUNCTION commands: * FUNCTION CREATE * FCALL * FCALL_RO * FUNCTION DELETE * FUNCTION KILL * FUNCTION INFO 2. Register engine In addition, this commit introduce the first engine that uses the Redis Function capabilities, the Lua engine.
-
- 01 Dec, 2021 4 commits
-
-
meir@redislabs.com authored
The functionality was moved to script_lua.c under callFunction function. Its purpose is to call the Lua function already located on the top of the Lua stack. Used the new function on eval.c to invoke Lua code. The function will also be used to invoke Lua code on the Lua engine.
-
meir@redislabs.com authored
Script unit is a new unit located on script.c. Its purpose is to provides an API for functions (and eval) to interact with Redis. Interaction includes mostly executing commands, but also functionalities like calling Redis back on long scripts or check if the script was killed. The interaction is done using a scriptRunCtx object that need to be created by the user and initialized using scriptPrepareForRun. Detailed list of functionalities expose by the unit: 1. Calling commands (including all the validation checks such as acl, cluster, read only run, ...) 2. Set Resp 3. Set Replication method (AOF/REPLICATION/NONE) 4. Call Redis back to on long running scripts to allow Redis reply to clients and perform script kill The commit introduce the new unit and uses it on eval commands to interact with Redis.
-
meir@redislabs.com authored
The following variable was renamed: 1. lua_caller -> script_caller 2. lua_time_limit -> script_time_limit 3. lua_timedout -> script_timedout 4. lua_oom -> script_oom 5. lua_disable_deny_script -> script_disable_deny_script 6. in_eval -> in_script The following variables was moved to lctx under eval.c 1. lua 2. lua_client 3. lua_cur_script 4. lua_scripts 5. lua_scripts_mem 6. lua_replicate_commands 7. lua_write_dirty 8. lua_random_dirty 9. lua_multi_emitted 10. lua_repl 11. lua_kill 12. lua_time_start 13. lua_time_snapshot This commit is in a low risk of introducing any issues and it is just moving varibales around and not changing any logic.
-
meir@redislabs.com authored
This commit is only move code around without changing it. The reason behind this is to make review process easier by allowing the reviewer to simply ignore all code movements. changes: 1. rename scripting.c to eval.c 2. introduce and new file, script_lua.c, and move parts of Lua functionality to this new file. script_lua.c will eventually contains the shared code between legacy lua and lua engine. This commit does not compiled on purpose. Its only purpose is to move code and rename files.
-
- 30 Nov, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
The issue can only happened with a bad Lua script that claims to return a big number while actually return data which is not a big number (contains chars that are not digits). Such thing will not cause an issue unless the big number value contains `\r\n` and then it messes the resp3 structure. The fix changes all the appearances of '\r\n' with spaces. Such an issue can also happened on simple string or error replies but those already handle it the same way this PR does (replace `\r\n` with spaces). Other replies type are not vulnerable to this issue because they are not counting on free text that is terminated with `\r\n` (either it contains the bulk length like string reply or they are typed reply that can not inject free text like boolean or number). The issue only exists on unstable branch, big number reply on Lua script was not yet added to any official release.
-
- 28 Nov, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
This commit 0f8b634c (CVE-2021-32626 released in 6.2.6, 6.0.16, 5.0.14) fixes an invalid memory write issue by using `lua_checkstack` API to make sure the Lua stack is not overflow. This fix was added on 3 places: 1. `luaReplyToRedisReply` 2. `ldbRedis` 3. `redisProtocolToLuaType` On the first 2 functions, `lua_checkstack` is handled gracefully while the last is handled with an assert and a statement that this situation can not happened (only with misbehave module): > the Redis reply might be deep enough to explode the LUA stack (notice that currently there is no such command in Redis that returns such a nested reply, but modules might do it) The issue that was discovered is that user arguments is also considered part of the stack, and so the following script (for example) make the assertion reachable: ``` local a = {} for i=1,7999 do a[i] = 1 end return redis.call("lpush", "l", unpack(a)) ``` This is a regression because such a script would have worked before and now its crashing Redis. The solution is to clear the function arguments from the Lua stack which makes the original assumption true and the assertion unreachable.
-
- 04 Nov, 2021 1 commit
-
-
Eduardo Semprebon authored
For diskless replication in swapdb mode, considering we already spend replica memory having a backup of current db to restore in case of failure, we can have the following benefits by instead swapping database only in case we succeeded in transferring db from master: - Avoid `LOADING` response during failed and successful synchronization for cases where the replica is already up and running with data. - Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping. - This could be implemented also for disk replication with similar benefits if consumers are willing to spend the extra memory usage. General notes: - The concept of `backupDb` becomes `tempDb` for clarity. - Async loading mode will only kick in if the replica is syncing from a master that has the same repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline. - New property in INFO: `async_loading` to differentiate from the blocking loading - Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db and the tempDb that is passed around. - Because this is affecting replicas only, we assume that if they are not readonly and write commands during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET here anyways to avoid complications. Considerations for review: - We have many cases where server.loading flag is used and even though I tried my best, there may be cases where async_loading should be checked as well and cases where it shouldn't (would require very good understanding of whole code) - Several places that had different behavior depending on the loading flag where actually meant to just handle commands coming from the AOF client differently than ones coming from real clients, changed to check CLIENT_ID_AOF instead. **Additional for Release Notes** - Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't contribute on triggering next database SAVE - New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING - Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event. Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED, ABORTED and COMPLETED. - New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions to allow modules to declare they support the diskless replication with async loading (when absent, we fall back to disk-based loading). Co-authored-by:
Eduardo Semprebon <edus@saxobank.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 24 Oct, 2021 1 commit
-
-
Oran Agra authored
I moved a bunch of stats in redisFork to be executed only on successful fork, since they seem wrong to be done when it failed. I guess when fork fails it does that immediately, no latency spike.
-
- 20 Oct, 2021 1 commit
-
-
guybe7 authored
## Intro The purpose is to allow having different flags/ACL categories for subcommands (Example: CONFIG GET is ok-loading but CONFIG SET isn't) We create a small command table for every command that has subcommands and each subcommand has its own flags, etc. (same as a "regular" command) This commit also unites the Redis and the Sentinel command tables ## Affected commands CONFIG Used to have "admin ok-loading ok-stale no-script" Changes: 1. Dropped "ok-loading" in all except GET (this doesn't change behavior since there were checks in the code doing that) XINFO Used to have "read-only random" Changes: 1. Dropped "random" in all except CONSUMERS XGROUP Used to have "write use-memory" Changes: 1. Dropped "use-memory" in all except CREATE and CREATECONSUMER COMMAND No changes. MEMORY Used to have "random read-only" Changes: 1. Dropped "random" in PURGE and USAGE ACL Used to have "admin no-script ok-loading ok-stale" Changes: 1. Dropped "admin" in WHOAMI, GENPASS, and CAT LATENCY No changes. MODULE No changes. SLOWLOG Used to have "admin random ok-loading ok-stale" Changes: 1. Dropped "random" in RESET OBJECT Used to have "read-only random" Changes: 1. Dropped "random" in ENCODING and REFCOUNT SCRIPT Used to have "may-replicate no-script" Changes: 1. Dropped "may-replicate" in all except FLUSH and LOAD CLIENT Used to have "admin no-script random ok-loading ok-stale" Changes: 1. Dropped "random" in all except INFO and LIST 2. Dropped "admin" in ID, TRACKING, CACHING, GETREDIR, INFO, SETNAME, GETNAME, and REPLY STRALGO No changes. PUBSUB No changes. CLUSTER Changes: 1. Dropped "admin in countkeysinslots, getkeysinslot, info, nodes, keyslot, myid, and slots SENTINEL No changes. (note that DEBUG also fits, but we decided not to convert it since it's for debugging and anyway undocumented) ## New sub-command This commit adds another element to the per-command output of COMMAND, describing the list of subcommands, if any (in the same structure as "regular" commands) Also, it adds a new subcommand: ``` COMMAND LIST [FILTERBY (MODULE <module-name>|ACLCAT <cat>|PATTERN <pattern>)] ``` which returns a set of all commands (unless filters), but excluding subcommands. ## Module API A new module API, RM_CreateSubcommand, was added, in order to allow module writer to define subcommands ## ACL changes: 1. Now, that each subcommand is actually a command, each has its own ACL id. 2. The old mechanism of allowed_subcommands is redundant (blocking/allowing a subcommand is the same as blocking/allowing a regular command), but we had to keep it, to support the widespread usage of allowed_subcommands to block commands with certain args, that aren't subcommands (e.g. "-select +select|0"). 3. I have renamed allowed_subcommands to allowed_firstargs to emphasize the difference. 4. Because subcommands are commands in ACL too, you can now use "-" to block subcommands (e.g. "+client -client|kill"), which wasn't possible in the past. 5. It is also possible to use the allowed_firstargs mechanism with subcommand. For example: `+config -config|set +config|set|loglevel` will block all CONFIG SET except for setting the log level. 6. All of the ACL changes above required some amount of refactoring. ## Misc 1. There are two approaches: Either each subcommand has its own function or all subcommands use the same function, determining what to do according to argv[0]. For now, I took the former approaches only with CONFIG and COMMAND, while other commands use the latter approach (for smaller blamelog diff). 2. Deleted memoryGetKeys: It is no longer needed because MEMORY USAGE now uses the "range" key spec. 4. Bugfix: GETNAME was missing from CLIENT's help message. 5. Sentinel and Redis now use the same table, with the same function pointer. Some commands have a different implementation in Sentinel, so we redirect them (these are ROLE, PUBLISH, and INFO). 6. Command stats now show the stats per subcommand (e.g. instead of stats just for "config" you will have stats for "config|set", "config|get", etc.) 7. It is now possible to use COMMAND directly on subcommands: COMMAND INFO CONFIG|GET (The pipeline syntax was inspired from ACL, and can be used in functions lookupCommandBySds and lookupCommandByCString) 8. STRALGO is now a container command (has "help") ## Breaking changes: 1. Command stats now show the stats per subcommand (see (5) above)
-
- 04 Oct, 2021 2 commits
-
-
Meir Shpilraien (Spielrein) authored
When LUA call our C code, by default, the LUA stack has room for 10 elements. In most cases, this is more than enough but sometimes it's not and the caller must verify the LUA stack size before he pushes elements. On 3 places in the code, there was no verification of the LUA stack size. On specific inputs this missing verification could have lead to invalid memory write: 1. On 'luaReplyToRedisReply', one might return a nested reply that will explode the LUA stack. 2. On 'redisProtocolToLuaType', the Redis reply might be deep enough to explode the LUA stack (notice that currently there is no such command in Redis that returns such a nested reply, but modules might do it) 3. On 'ldbRedis', one might give a command with enough arguments to explode the LUA stack (all the arguments will be pushed to the LUA stack) This commit is solving all those 3 issues by calling 'lua_checkstack' and verify that there is enough room in the LUA stack to push elements. In case 'lua_checkstack' returns an error (there is not enough room in the LUA stack and it's not possible to increase the stack), we will do the following: 1. On 'luaReplyToRedisReply', we will return an error to the user. 2. On 'redisProtocolToLuaType' we will exit with panic (we assume this scenario is rare because it can only happen with a module). 3. On 'ldbRedis', we return an error.
-
Oran Agra authored
The protocol parsing on 'ldbReplParseCommand' (LUA debugging) Assumed protocol correctness. This means that if the following is given: *1 $100 test The parser will try to read additional 94 unallocated bytes after the client buffer. This commit fixes this issue by validating that there are actually enough bytes to read. It also limits the amount of data that can be sent by the debugger client to 1M so the client will not be able to explode the memory. Co-authored-by:
meir@redislabs.com <meir@redislabs.com>
-
- 23 Sep, 2021 1 commit
-
-
YaacovHazan authored
This commit introduced a new flag to the RM_Call: 'C' - Check if the command can be executed according to the ACLs associated with it. Also, three new API's added to check if a command, key, or channel can be executed or accessed by a user, according to the ACLs associated with it. - RM_ACLCheckCommandPerm - RM_ACLCheckKeyPerm - RM_ACLCheckChannelPerm The user for these API's is a RedisModuleUser object, that for a Module user returned by the RM_CreateModuleUser API, or for a general ACL user can be retrieved by these two new API's: - RM_GetCurrentUserName - Retrieve the user name of the client connection behind the current context. - RM_GetModuleUserFromUserName - Get a RedisModuleUser from a user name As a result of getting a RedisModuleUser from name, it can now also access the general ACL users (not just ones created by the module). This mean the already existing API RM_SetModuleUserACL(), can be used to change the ACL rules for such users.
-
- 05 Aug, 2021 1 commit
-
-
yoav-steinberg authored
Reduce dict struct memory overhead on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin. summary of changes: - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below). - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c) - Eliminate the `sizemask` field, can be calculated from size when needed. - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8. Interface change: pass dict pointer to dict type call back functions. This is instead of passing the removed privdata field. In the future if we'd like to have private data in the callbacks we can extract it from the dict type. We can extend dictType to include a custom dict struct allocator and use it to allocate more data at the end of the dict struct. This data can then be used to store private data later acccessed by the callbacks.
-
- 04 Aug, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
## Current state 1. Lua has its own parser that handles parsing `reds.call` replies and translates them to Lua objects that can be used by the user Lua code. The parser partially handles resp3 (missing big number, verbatim, attribute, ...) 2. Modules have their own parser that handles parsing `RM_Call` replies and translates them to RedisModuleCallReply objects. The parser does not support resp3. In addition, in the future, we want to add Redis Function (#8693) that will probably support more languages. At some point maintaining so many parsers will stop scaling (bug fixes and protocol changes will need to be applied on all of them). We will probably end up with different parsers that support different parts of the resp protocol (like we already have today with Lua and modules) ## PR Changes This PR attempt to unified the reply parsing of Lua and modules (and in the future Redis Function) by introducing a new parser unit (`resp_parser.c`). The new parser handles parsing the reply and calls different callbacks to allow the users (another unit that uses the parser, i.e, Lua, modules, or Redis Function) to analyze the reply. ### Lua API Additions The code that handles reply parsing on `scripting.c` was removed. Instead, it uses the resp_parser to parse and create a Lua object out of the reply. As mentioned above the Lua parser did not handle parsing big numbers, verbatim, and attribute. The new parser can handle those and so Lua also gets it for free. Those are translated to Lua objects in the following way: 1. Big Number - Lua table `{'big_number':'<str representation for big number>'}` 2. Verbatim - Lua table `{'verbatim_string':{'format':'<verbatim format>', 'string':'<verbatim string value>'}}` 3. Attribute - currently ignored and not expose to the Lua parser, another issue will be open to decide how to expose it. Tests were added to check resp3 reply parsing on Lua ### Modules API Additions The reply parsing code on `module.c` was also removed and the new resp_parser is used instead. In addition, the RedisModuleCallReply was also extracted to a separate unit located on `call_reply.c` (in the future, this unit will also be used by Redis Function). A nice side effect of unified parsing is that modules now also support resp3. Resp3 can be enabled by giving `3` as a parameter to the fmt argument of `RM_Call`. It is also possible to give `0`, which will indicate an auto mode. i.e, Redis will automatically chose the reply protocol base on the current client set on the RedisModuleCtx (this mode will mostly be used when the module want to pass the reply to the client as is). In addition, the following RedisModuleAPI were added to allow analyzing resp3 replies: * New RedisModuleCallReply types: * `REDISMODULE_REPLY_MAP` * `REDISMODULE_REPLY_SET` * `REDISMODULE_REPLY_BOOL` * `REDISMODULE_REPLY_DOUBLE` * `REDISMODULE_REPLY_BIG_NUMBER` * `REDISMODULE_REPLY_VERBATIM_STRING` * `REDISMODULE_REPLY_ATTRIBUTE` * New RedisModuleAPI: * `RedisModule_CallReplyDouble` - getting double value from resp3 double reply * `RedisModule_CallReplyBool` - getting boolean value from resp3 boolean reply * `RedisModule_CallReplyBigNumber` - getting big number value from resp3 big number reply * `RedisModule_CallReplyVerbatim` - getting format and value from resp3 verbatim reply * `RedisModule_CallReplySetElement` - getting element from resp3 set reply * `RedisModule_CallReplyMapElement` - getting key and value from resp3 map reply * `RedisModule_CallReplyAttribute` - getting a reply attribute * `RedisModule_CallReplyAttributeElement` - getting key and value from resp3 attribute reply * New context flags: * `REDISMODULE_CTX_FLAGS_RESP3` - indicate that the client is using resp3 Tests were added to check the new RedisModuleAPI ### Modules API Changes * RM_ReplyWithCallReply might return REDISMODULE_ERR if the given CallReply is in resp3 but the client expects resp2. This is not a breaking change because in order to get a resp3 CallReply one needs to specifically specify `3` as a parameter to the fmt argument of `RM_Call` (as mentioned above). Tests were added to check this change ### More small Additions * Added `debug set-disable-deny-scripts` that allows to turn on and off the commands no-script flag protection. This is used by the Lua resp3 tests so it will be possible to run `debug protocol` and check the resp3 parsing code. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
- 18 Jul, 2021 1 commit
-
-
Paul Kulchenko authored
-
- 10 Jun, 2021 1 commit
-
-
Binbin authored
This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes. This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything, but at least it is also unlikely to cause false positives. Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use. Here's a summary of other changes: 1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces). 2. Outdated function / variable / argument names in comments 3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString. 4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751 5. Some outdated https link URLs. 6. Fix some outdated comment. Such as: - In README: about the rdb, we used to said create a `thread`, change to `process` - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey) - notifyKeyspaceEvent fucntion comment (add type arg) - Some others minor fix in comment (Most of them are incorrectly quoted by variable names) 7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
-
- 08 Jun, 2021 1 commit
-
-
Wang Yuan authored
When we allocate a client struct with 16k reply buffer, the allocator we may give us 20K, This commit makes use of that extra space. Additionally, it tries to store whatever it can from the reply into the static 'buf' before allocating a new node for the reply list.
-
- 19 May, 2021 1 commit
-
-
Madelyn Olson authored
Redact commands that include sensitive data from slowlog and monitor
-
- 13 May, 2021 1 commit
-
-
Raghav Muddur authored
* EVALSHA_RO and EVAL_RO Commands Added new readonly versions of EVAL and EVALSHA.
-
- 22 Apr, 2021 1 commit
-
-
zyxwvu Shi authored
This prevents a case where NTP moves the system clock forward resulting in a false detection of a busy script. Signed-off-by:
zyxwvu Shi <i@shiyc.cn>
-
- 26 Mar, 2021 1 commit
-
-
Huang Zhw authored
Add publish channel permissions check in processCommand. processCommand didn't check publish channel permissions, so we can queue a publish command in a transaction. But when exec the transaction, it will fail with -NOPERM. We also union keys/commands/channels permissions check togegher in ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in publishCommand/subscribeCommand/psubscribeCommand. Always check permissions in processCommand/execCommand/ luaRedisGenericCommand.
-
- 17 Mar, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
pcall function runs another LUA function in protected mode, this means that any error will be caught by this function and will not stop the LUA execution. The script kill mechanism uses error to stop the running script. Scripts that uses pcall can catch the error raise by the script kill mechanism, this will cause a script like this to be unkillable: local f = function() while 1 do redis.call('ping') end end while 1 do pcall(f) end The fix is, when we want to kill the script, we set the hook function to be invoked after each line. This will promise that the execution will get another error before it is able to enter the pcall function again.
-
- 15 Mar, 2021 1 commit
-
-
guybe7 authored
1. moduleReplicateMultiIfNeeded should use server.in_eval like moduleHandlePropagationAfterCommandCallback 2. server.in_eval could have been set to 1 and not reset back to 0 (a lot of missed early-exits after in_eval is already 1) Note: The new assertions in processCommand cover (2) and I added two module tests to cover (1) Implications: If an EVAL that failed (and thus left server.in_eval=1) runs before a module command that replicates, the replication stream will contain MULTI (because moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL at this point) but not EXEC (because server.in_eval==1) This only affects modules as module.c the only user of server.in_eval. Affects versions 6.2.0, 6.2.1
-
- 24 Feb, 2021 2 commits
- 09 Feb, 2021 1 commit
-
-
Madelyn Olson authored
Moved most static strings into the shared structure
-
- 15 Jan, 2021 1 commit
-
-
Yang Bodong authored
Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258) * Adds ASYNC and SYNC arguments to SCRIPT FLUSH * Adds SYNC argument to FLUSHDB and FLUSHALL * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH. the new behavior is as follows: * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the value of lazyfree-lazy-user-flush. * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner. * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
-
- 05 Jan, 2021 1 commit
-
-
Oran Agra authored
When a Lua script returns a map to redis (a feature which was added in redis 6 together with RESP3), it would have returned the value first and the key second. If the client was using RESP2, it was getting them out of order, and if the client was in RESP3, it was getting a map of value => key. This was happening regardless of the Lua script using redis.setresp(3) or not. This also affects a case where the script was returning a map which it got from from redis by doing something like: redis.setresp(3); return redis.call() This fix is a breaking change for redis 6.0 users who happened to rely on the wrong order (either ones that used redis.setresp(3), or ones that returned a map explicitly). This commit also includes other two changes in the tests: 1. The test suite now handles RESP3 maps as dicts rather than nested lists 2. Remove some redundant (duplicate) tests from tracking.tcl
-
- 04 Jan, 2021 1 commit
-
-
Itamar Haber authored
* man-like consistent long formatting * Uppercases commands, subcommands and options * Adds 'HELP' to HELP for all * Lexicographical order * Uses value notation and other .md likeness * Moves const char *help to top * Keeps it under 80 chars * Misc help typos, consistent conjuctioning (i.e return and not returns) * Uses addReplySubcommandSyntaxError(c) all over Signed-off-by:
Itamar Haber <itamar@redislabs.com>
-
- 27 Dec, 2020 1 commit
-
-
Oran Agra authored
Recently efaf09ee started using addReplyErrorSds in place of addReplySds the later takes ownership of the string but the former did not. This introduced memory leaks when a script returns an error to redis, and also in clusterRedirectClient (two new usages of addReplyErrorSds which was mostly unused till now. This commit chagnes two thanks. 1. change addReplyErrorSds to take ownership of the error string. 2. scripting.c doesn't actually need to use addReplyErrorSds, it's a perfect match for addReplyErrorFormat (replaces newlines with spaces)
-
- 24 Dec, 2020 1 commit
-
-
Madelyn Olson authored
Properly throw errors for invalid replication stream and support https://github.com/redis/redis/pull/8217
-
- 22 Dec, 2020 1 commit
-
-
Oran Agra authored
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
-
- 06 Dec, 2020 1 commit
-
-
guybe7 authored
One way this was happening is when a module issued an RM_Call which would inject MULTI. If the module command that does that was itself issued by something else that already did added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI. In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't the right indication that we need to propagate MULTI or not, because on a nested calls (possibly a module action called by a keyspace event of another module action), these flags aren't retained / reflected. instead there's now a global propagate_in_transaction flag for that. in addition to that, we now have a global in_eval and in_exec flags, to serve the flags of RM_GetContextFlags, since their dependence on the current client is wrong for the same reasons mentioned above.
-
- 01 Dec, 2020 1 commit
-
-
Itamar Haber authored
Fixes #7923. This PR appropriates the special `&` symbol (because `@` and `*` are taken), followed by a literal value or pattern for describing the Pub/Sub patterns that an ACL user can interact with. It is similar to the existing key patterns mechanism in function (additive) and implementation (copy-pasta). It also adds the allchannels and resetchannels ACL keywords, naturally. The default user is given allchannels permissions, whereas new users get whatever is defined by the acl-pubsub-default configuration directive. For backward compatibility in 6.2, the default of this directive is allchannels but this is likely to be changed to resetchannels in the next major version for stronger default security settings. Unless allchannels is set for the user, channel access permissions are checked as follows : * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the argumentative channel name(s) exists for the user. * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument literally exist(s) in the user's list. Such failures are logged to the ACL log. Runtime changes to channel permissions for a user with existing subscribing clients cause said clients to disconnect unless the new permissions permit the connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be disconnected. Notes/questions: * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons for touching them.
-
- 17 Nov, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 20 Sep, 2020 1 commit
-
-
Oran Agra authored
-