- 12 Dec, 2022 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Fix #11030, use lua_rawget to avoid triggering metatables. #11030 shows how return `_G` from the Lua script (either function or eval), cause the Lua interpreter to Panic and the Redis processes to exit with error code 1. Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable such that the metatable raises an error. The following example demonstrate the issue: ``` 127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0 Error: Server closed the connection ``` ``` PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo') ``` The Lua panic happened because when returning the result to the client, Redis needs to introspect the returning table and transform the table into a resp. In order to scan the table, Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error. This code is not running inside `pcall` (Lua protected call), so raising an error causes the Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1. Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable that raises error when trying to fetch a none existing key. ### Solution Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget` that simply return the value from the table without triggering any metatable logic. This is promised not to raise and error. The downside of this solution is that it might be considered as breaking change, if someone rely on metatable in the returned value. An alternative solution is to wrap this entire logic with `pcall` (Lua protected call), this alternative require a much bigger refactoring. ### Back Porting The same fix will work on older versions as well (6.2, 6.0). Notice that on those version, the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done from inside the script itself. ### Tests Tests was added the verify the fix (cherry picked from commit 020e046b)
-
- 06 Dec, 2022 1 commit
-
-
Meir Shpilraien (Spielrein) authored
On v6.2.7 a new mechanism was added to Lua scripts that allows filtering the globals of the Lua interpreter. This mechanism was added on the following commit: https://github.com/redis/redis/commit/11b602fbf8f9cdf8fc741c24625ab6287ab998a9 One of the globals that was filtered out was `__redis__compare_helper`. This global was missed and was not added to the allow list or to the deny list. This is why we get the following warning when Redis starts: `A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.` After investigating the git blame log, the conclusion is that `__redis__compare_helper` is no longer needed, the PR deletes this function, and fixes the warning. Detailed Explanation: `__redis__compare_helper` was added on this commit: https://github.com/redis/redis/commit/2c861050c1 Its purpose is to sort the replies of `SORT` command when script replication is enable and keep the replies deterministic and avoid primary and replica synchronization issues. On `SORT` command, there was a need for special compare function that are able to compare boolean values. The need to sort the `SORT` command reply was removed on this commit: https://github.com/redis/redis/commit/36741b2c818a95e8ef167818271614ee6b1bc414 The sorting was moved to be part of the `SORT` command and there was not longer a need to sort it on the Lua interpreter. The commit made `__redis__compare_helper` a dead code but did not deleted it.
-
- 27 Apr, 2022 5 commits
-
-
Oran Agra authored
-
Meir Shpilraien (Spielrein) authored
This commit 0f8b634c (CVE-2021-32626 released in 6.2.6, 6.0.16, 5.0.14) fixes an invalid memory write issue by using `lua_checkstack` API to make sure the Lua stack is not overflow. This fix was added on 3 places: 1. `luaReplyToRedisReply` 2. `ldbRedis` 3. `redisProtocolToLuaType` On the first 2 functions, `lua_checkstack` is handled gracefully while the last is handled with an assert and a statement that this situation can not happened (only with misbehave module): > the Redis reply might be deep enough to explode the LUA stack (notice that currently there is no such command in Redis that returns such a nested reply, but modules might do it) The issue that was discovered is that user arguments is also considered part of the stack, and so the following script (for example) make the assertion reachable: ``` local a = {} for i=1,7999 do a[i] = 1 end return redis.call("lpush", "l", unpack(a)) ``` This is a regression because such a script would have worked before and now its crashing Redis. The solution is to clear the function arguments from the Lua stack which makes the original assumption true and the assertion unreachable. (cherry picked from commit 6b0b04f1)
-
meir authored
The allow list is done by setting a metatable on the global table before initializing any library. The metatable set the `__newindex` field to a function that check the allow list before adding the field to the table. Fields which is not on the allow list are simply ignored. After initialization phase is done we protect the global table and each table that might be reachable from the global table. For each table we also protect the table metatable if exists.
-
meir authored
Use the new `lua_enablereadonlytable` Lua API to protect the global tables of evals scripts. The implemetation is easy, we simply call `lua_enablereadonlytable` on the global table to turn it into a readonly table.
-
meir authored
Today, Redis wrap the user Lua code with a Lua function. For example, assuming the user code is: ``` return redis.call('ping') ``` The actual code that would have sent to the Lua interpreter was: ``` f_b3a02c833904802db9c34a3cf1292eee3246df3c() return redis.call('ping') end ``` The wraped code would have been saved on the global dictionary with the following name: `f_<script sha>` (in our example `f_b3a02c833904802db9c34a3cf1292eee3246df3c`). This approach allows one user to easily override the implementation a another user code, example: ``` f_b3a02c833904802db9c34a3cf1292eee3246df3c = function() return 'hacked' end ``` Running the above code will cause `evalsha b3a02c833904802db9c34a3cf1292eee3246df3c 0` to return hacked although it should have returned `pong`. Another disadventage is that Redis basically runs code on the loading (compiling) phase without been aware of it. User can do code injection like this: ``` return 1 end <run code on compling phase> function() return 1 ``` The wraped code will look like this and the entire `<run code on compling phase>` block will run outside of eval or evalsha context: ``` f_<sha>() return 1 end <run code on compling phase> function() return 1 end ```
-
- 11 Apr, 2022 1 commit
-
-
Vo Trong Phuc authored
There was no check min-slave-* config when evaluating Lua script. Add check enough good slaves for write command when evaluating scripts. Co-authored-by:Phuc. Vo Trong <phucvt@vng.com.vn>
-
- 04 Oct, 2021 2 commits
-
-
meir@redislabs.com authored
When LUA call our C code, by default, the LUA stack has room for 20 elements. In most cases, this is more than enough but sometimes it's not and the caller must verify the LUA stack size before he pushes elements. On 3 places in the code, there was no verification of the LUA stack size. On specific inputs this missing verification could have lead to invalid memory write: 1. On 'luaReplyToRedisReply', one might return a nested reply that will explode the LUA stack. 2. On 'redisProtocolToLuaType', the Redis reply might be deep enough to explode the LUA stack (notice that currently there is no such command in Redis that returns such a nested reply, but modules might do it) 3. On 'ldbRedis', one might give a command with enough arguments to explode the LUA stack (all the arguments will be pushed to the LUA stack) This commit is solving all those 3 issues by calling 'lua_checkstack' and verify that there is enough room in the LUA stack to push elements. In case 'lua_checkstack' returns an error (there is not enough room in the LUA stack and it's not possible to increase the stack), we will do the following: 1. On 'luaReplyToRedisReply', we will return an error to the user. 2. On 'redisProtocolToLuaType' we will exit with panic (we assume this scenario is rare because it can only happen with a module). 3. On 'ldbRedis', we return an error.
-
meir@redislabs.com authored
The protocol parsing on 'ldbReplParseCommand' (LUA debugging) Assumed protocol correctness. This means that if the following is given: *1 $100 test The parser will try to read additional 94 unallocated bytes after the client buffer. This commit fixes this issue by validating that there are actually enough bytes to read. It also limits the amount of data that can be sent by the debugger client to 1M so the client will not be able to explode the memory.
-
- 01 Jun, 2021 1 commit
-
-
Madelyn Olson authored
Redact commands that include sensitive data from slowlog and monitor (cherry picked from commit a59e75a4)
-
- 03 May, 2021 1 commit
-
-
zyxwvu Shi authored
This prevents a case where NTP moves the system clock forward resulting in a false detection of a busy script. Signed-off-by:
zyxwvu Shi <i@shiyc.cn> (cherry picked from commit f61c37ce)
-
- 26 Mar, 2021 1 commit
-
-
Huang Zhw authored
Add publish channel permissions check in processCommand. processCommand didn't check publish channel permissions, so we can queue a publish command in a transaction. But when exec the transaction, it will fail with -NOPERM. We also union keys/commands/channels permissions check togegher in ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in publishCommand/subscribeCommand/psubscribeCommand. Always check permissions in processCommand/execCommand/ luaRedisGenericCommand.
-
- 17 Mar, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
pcall function runs another LUA function in protected mode, this means that any error will be caught by this function and will not stop the LUA execution. The script kill mechanism uses error to stop the running script. Scripts that uses pcall can catch the error raise by the script kill mechanism, this will cause a script like this to be unkillable: local f = function() while 1 do redis.call('ping') end end while 1 do pcall(f) end The fix is, when we want to kill the script, we set the hook function to be invoked after each line. This will promise that the execution will get another error before it is able to enter the pcall function again.
-
- 15 Mar, 2021 1 commit
-
-
guybe7 authored
1. moduleReplicateMultiIfNeeded should use server.in_eval like moduleHandlePropagationAfterCommandCallback 2. server.in_eval could have been set to 1 and not reset back to 0 (a lot of missed early-exits after in_eval is already 1) Note: The new assertions in processCommand cover (2) and I added two module tests to cover (1) Implications: If an EVAL that failed (and thus left server.in_eval=1) runs before a module command that replicates, the replication stream will contain MULTI (because moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL at this point) but not EXEC (because server.in_eval==1) This only affects modules as module.c the only user of server.in_eval. Affects versions 6.2.0, 6.2.1
-
- 24 Feb, 2021 2 commits
- 09 Feb, 2021 1 commit
-
-
Madelyn Olson authored
Moved most static strings into the shared structure
-
- 15 Jan, 2021 1 commit
-
-
Yang Bodong authored
Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258) * Adds ASYNC and SYNC arguments to SCRIPT FLUSH * Adds SYNC argument to FLUSHDB and FLUSHALL * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH. the new behavior is as follows: * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the value of lazyfree-lazy-user-flush. * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner. * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
-
- 05 Jan, 2021 1 commit
-
-
Oran Agra authored
When a Lua script returns a map to redis (a feature which was added in redis 6 together with RESP3), it would have returned the value first and the key second. If the client was using RESP2, it was getting them out of order, and if the client was in RESP3, it was getting a map of value => key. This was happening regardless of the Lua script using redis.setresp(3) or not. This also affects a case where the script was returning a map which it got from from redis by doing something like: redis.setresp(3); return redis.call() This fix is a breaking change for redis 6.0 users who happened to rely on the wrong order (either ones that used redis.setresp(3), or ones that returned a map explicitly). This commit also includes other two changes in the tests: 1. The test suite now handles RESP3 maps as dicts rather than nested lists 2. Remove some redundant (duplicate) tests from tracking.tcl
-
- 04 Jan, 2021 1 commit
-
-
Itamar Haber authored
* man-like consistent long formatting * Uppercases commands, subcommands and options * Adds 'HELP' to HELP for all * Lexicographical order * Uses value notation and other .md likeness * Moves const char *help to top * Keeps it under 80 chars * Misc help typos, consistent conjuctioning (i.e return and not returns) * Uses addReplySubcommandSyntaxError(c) all over Signed-off-by:Itamar Haber <itamar@redislabs.com>
-
- 27 Dec, 2020 1 commit
-
-
Oran Agra authored
Recently efaf09ee started using addReplyErrorSds in place of addReplySds the later takes ownership of the string but the former did not. This introduced memory leaks when a script returns an error to redis, and also in clusterRedirectClient (two new usages of addReplyErrorSds which was mostly unused till now. This commit chagnes two thanks. 1. change addReplyErrorSds to take ownership of the error string. 2. scripting.c doesn't actually need to use addReplyErrorSds, it's a perfect match for addReplyErrorFormat (replaces newlines with spaces)
-
- 24 Dec, 2020 1 commit
-
-
Madelyn Olson authored
Properly throw errors for invalid replication stream and support https://github.com/redis/redis/pull/8217
-
- 22 Dec, 2020 1 commit
-
-
Oran Agra authored
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
-
- 06 Dec, 2020 1 commit
-
-
guybe7 authored
One way this was happening is when a module issued an RM_Call which would inject MULTI. If the module command that does that was itself issued by something else that already did added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI. In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't the right indication that we need to propagate MULTI or not, because on a nested calls (possibly a module action called by a keyspace event of another module action), these flags aren't retained / reflected. instead there's now a global propagate_in_transaction flag for that. in addition to that, we now have a global in_eval and in_exec flags, to serve the flags of RM_GetContextFlags, since their dependence on the current client is wrong for the same reasons mentioned above.
-
- 01 Dec, 2020 1 commit
-
-
Itamar Haber authored
Fixes #7923. This PR appropriates the special `&` symbol (because `@` and `*` are taken), followed by a literal value or pattern for describing the Pub/Sub patterns that an ACL user can interact with. It is similar to the existing key patterns mechanism in function (additive) and implementation (copy-pasta). It also adds the allchannels and resetchannels ACL keywords, naturally. The default user is given allchannels permissions, whereas new users get whatever is defined by the acl-pubsub-default configuration directive. For backward compatibility in 6.2, the default of this directive is allchannels but this is likely to be changed to resetchannels in the next major version for stronger default security settings. Unless allchannels is set for the user, channel access permissions are checked as follows : * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the argumentative channel name(s) exists for the user. * Calls t...
-
- 17 Nov, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 20 Sep, 2020 1 commit
-
-
Oran Agra authored
-
- 10 Sep, 2020 1 commit
-
-
Oran Agra authored
List of squashed commits or PRs =============================== commit 66801ea Author: hwware <wen.hui.ware@gmail.com> Date: Mon Jan 13 00:54:31 2020 -0500 typo fix in acl.c commit 46f55db Author: Itamar Haber <itamar@redislabs.com> Date: Sun Sep 6 18:24:11 2020 +0300 Updates a couple of comments Specifically: * RM_AutoMemory completed instead of pointing to docs * Updated link to custom type doc commit 61a2aa0 Author: xindoo <xindoo@qq.com> Date: Tue Sep 1 19:24:59 2020 +0800 Correct errors in code comments commit a5871d1 Author: yz1509 <pro-756@qq.com> Date: Tue Sep 1 18:36:06 2020 +0800 fix typos in module.c commit 41eede7 Author: bookug <bookug@qq.com> Date: Sat Aug 15 01:11:33 2020 +0800 docs: fix typos in comments commit c303c84 Author: lazy-snail <ws.niu@outlook.com> Date: Fri Aug 7 11:15:44 2020 +0800 fix spelling in redis.conf commit 1e...
-
- 03 Sep, 2020 1 commit
-
-
Oran Agra authored
During long running scripts or loading RDB/AOF, we may need to do some defragging. Since processEventsWhileBlocked is called periodically at unknown intervals, and many cron jobs either depend on run_with_period (including active defrag), or rely on being called at server.hz rate (i.e. active defrag knows ho much time to run by looking at server.hz), the whileBlockedCron may have to run a loop triggering the cron jobs in it (currently only active defrag) several times. Other changes: - Adding a test for defrag during aof loading. - Changing key-load-delay config to take negative values for fractions of a microsecond sleep
-
- 06 Apr, 2020 1 commit
-
-
mymilkbottles authored
-
- 04 Feb, 2020 2 commits
- 30 Jan, 2020 1 commit
-
-
antirez authored
-
- 27 Jan, 2020 1 commit
-
-
antirez authored
-
- 20 Jan, 2020 1 commit
-
-
srzhao authored
Checking OOM by `getMaxMemoryState` inside script might get different result with `freeMemoryIfNeededAndSafe` at script start, because lua stack and arguments also consume memory. This leads to memory `borderline` when memory grows near server.maxmemory: - `freeMemoryIfNeededAndSafe` at script start detects no OOM, no memory freed - `getMaxMemoryState` inside script detects OOM, script aborted We solve this 'borderline' issue by saving OOM state at script start to get stable lua OOM state. related to issue #6565 and #5250.
-
- 17 Dec, 2019 3 commits
-
-
Madelyn Olson authored
-
Madelyn Olson authored
-
Madelyn Olson authored
-
- 22 Nov, 2019 1 commit
-
-
zhaozhao.zz authored
-