- 12 Dec, 2022 14 commits
-
-
ranshid authored
When using cli to add node, there can potentially be a race condition in which all nodes presenting cluster state o.k even though the added node did not yet meet all cluster nodes. this adds another utility function to wait until all cluster nodes see the same cluster size (cherry picked from commit c0ce97fa)
-
Binbin authored
A timing issue like this was reported in freebsd daily CI: ``` *** [err]: Sanity test push cmd after resharding in tests/unit/cluster/cli.tcl Expected 'CLUSTERDOWN The cluster is down' to match '*MOVED*' ``` We additionally wait for each node to reach a consensus on the cluster state in wait_for_condition to avoid the cluster down error. The fix just like #10495, quoting madolson's comment: Cluster check just verifies the the config state is self-consistent, waiting for cluster_state to be okay is an independent check that all the nodes actually believe each other are healthy. At the same time i noticed that unit/moduleapi/cluster.tcl has an exact same test, may have the same problem, also modified it. (cherry picked from commit 5ce64ab0)
-
Harkrishn Patro authored
## Issue During the client input/output buffer processing, the memory usage is incrementally updated to keep track of clients going beyond a certain threshold `maxmemory-clients` to be evicted. However, this additional tracking activity leads to unnecessary CPU cycles wasted when no client-eviction is required. It is applicable in two cases. * `maxmemory-clients` is set to `0` which equates to no client eviction (applicable to all clients) * `CLIENT NO-EVICT` flag is set to `ON` which equates to a particular client not applicable for eviction. ## Solution * Disable client memory usage tracking during the read/write flow when `maxmemory-clients` is set to `0` or `client no-evict` is `on`. The memory usage is tracked only during the `clientCron` i.e. it gets periodically updated. * Cleanup the clients from the memory usage bucket when client eviction is disabled. * When the maxmemory-clients config is enabled or disabled at runtime, we immediately update the memory usage buckets for all clients (tested scanning 80000 took some 20ms) Benchmark shown that this can improve performance by about 5% in certain situations. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit c0267b3f)
-
filipe oliveira authored
This mechanism aims to reduce calls to malloc and free when preparing the arguments the script sends to redis commands. This is a mechanism was originally implemented in 48c49c48 and 4f686555 , and was removed in #10220 (thinking it's not needed and that it has no impact), but it now turns out it was wrong, and it indeed provides some 5% performance improvement. The implementation is a little bit too simplistic, it assumes consecutive calls use the same size in the same arg index, but that's arguably sufficient since it's only aimed at caching very small things. We could even consider always pre-allocating args to the full LUA_CMD_OBJCACHE_MAX_LEN (64 bytes) rather than the right size for the argument, that would increase the chance they'll be able to be re-used. But in some way this is already happening since we're using sdsalloc, which in turn uses s_malloc_usable and takes ownership of the full side of the allocation, so we are padded to the allocator bucket size. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
sundb <sundbcn@gmail.com> (cherry picked from commit 2d80cd78)
-
filipe oliveira authored
GEODIST used snprintf("%.4f") for the reply using addReplyDoubleDistance, which was slow. This PR optimizes it without breaking compatibility by following the approach of ll2string with some changes to match the use case of distance and precision. I.e. we multiply it by 10000 format it as an integer, and then add a decimal point. This can achieve about 35% increase in the achievable ops/sec. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit 61c85a2b)
-
DevineLiu authored
The cluster-announce-port/cluster-announce-bus-port/cluster-announce-tls-port should take effect at runtime Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com> (cherry picked from commit 25ffa79b)
-
Madelyn Olson authored
Both functions and eval are marked as "no-monitor", since we want to explicitly feed in the script command before the commands generated by the script. Note that we want this behavior generally, so that commands can redact arguments before being added to the monitor. (cherry picked from commit d136bf28)
-
uriyage authored
In moduleFireServerEvent we change the real client DB to 0 on freeClient in case the event is REDISMODULE_EVENT_CLIENT_CHANGE. It results in a crash if the client is blocked on a key on other than DB 0. The DB change is not necessary even for module-client, as we set its DB to 0 on either createClient or moduleReleaseTempClient. Co-authored-by:
Madelyn Olson <34459052+madolson@users.noreply.github.com> Co-authored-by:
Binbin <binloveplay1314@qq.com> (cherry picked from commit e4eb18b3)
-
sundb authored
In the module, we will reuse the list iterator entry for RM_ListDelete, but `listTypeDelete` will only update `quicklistEntry->zi` but not `quicklistEntry->node`, which will result in `quicklistEntry->node` pointing to a freed memory address if the quicklist node is deleted. This PR sync `key->u.list.index` and `key->u.list.entry` to list iterator after `RM_ListDelete`. This PR also optimizes the release code of the original list iterator. Co-authored-by:
Viktor Söderqvist <viktor@zuiderkwast.se> (cherry picked from commit 6dd21355)
-
C Charles authored
MIGTATE with AUTH that contains "keys" is getting wrong key names in migrateGetKeys, leads to ACL errors (#11253) When using the MIGRATE, with a destination Redis that has the user name or password set to the string "keys", Redis would have determine the wrong set of key names the command is gonna access. This lead to ACL returning wrong authentication result. Destination instance: ``` 127.0.0.1:6380> acl setuser default >keys OK 127.0.0.1:6380> acl setuser keys on nopass ~* &* +@all OK ``` Source instance: ``` 127.0.0.1:6379> set a 123 OK 127.0.0.1:6379> acl setuser cc on nopass ~a* +@all OK 127.0.0.1:6379> auth cc 1 OK 127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth keys keys a (error) NOPERM this user has no permissions to access one of the keys used as arguments 127.0.0.1:6379> migrate 127.0.0.1 6380 "" 0 1000 auth2 keys pswd keys a (error) NOPERM this user has no permissions to access one of the keys used as arguments ``` Using `acl dryrun` we know that the parameters of `auth` and `auth2` are mistaken for the `keys` option. ``` 127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth keys keys a "This user has no permissions to access the 'keys' key" 127.0.0.1:6379> acl dryrun cc migrate whatever whatever "" 0 1000 auth2 keys pswd keys a "This user has no permissions to access the 'pswd' key" ``` Fix the bug by editing db.c/migrateGetKeys function, which finds the `keys` option and all the keys following. (cherry picked from commit 9ab873d9)
-
Meir Shpilraien (Spielrein) authored
As mentioned on docs, `RM_ResetDataset` Performs similar operation to FLUSHALL. As FLUSHALL do not clean the function, `RM_ResetDataset` should not clean the functions as well. (cherry picked from commit d2ad01ab)
-
guybe7 authored
The original idea behind auto-setting the default (first,last,step) spec was to use the most "open" flags when the user didn't provide any key-spec flags information. While the above idea is a good approach, it really makes no sense to set CMD_KEY_VARIABLE_FLAGS if the user didn't provide the getkeys-api flag: in this case there's not way to retrieve these variable flags, so what's the point? Internally in redis there was code to ignore this already, so this fix doesn't change redis's behavior, it only affects the output of COMMAND command. (cherry picked from commit 3330ea18)
-
Shaya Potter authored
Adds a number of user management/ACL validaiton/command execution functions to improve a Redis module's ability to enforce ACLs correctly and easily. * RM_SetContextUser - sets a RedisModuleUser on the context, which RM_Call will use to both validate ACLs (if requested and set) as well as assign to the client so that scripts executed via RM_Call will have proper ACL validation. * RM_SetModuleUserACLString - Enables one to pass an entire ACL string, not just a single OP and have it applied to the user * RM_GetModuleUserACLString - returns a stringified version of the user's ACL (same format as dump and list). Contains an optimization to cache the stringified version until the underlying ACL is modified. * Slightly re-purpose the "C" flag to RM_Call from just being about ACL check before calling the command, to actually running the command with the right user, so that it also affects commands inside EVAL scripts. see #11231 (cherry picked from commit 6e993a5d)
-
- 21 Sep, 2022 12 commits
-
-
Oran Agra authored
Executing an XAUTOCLAIM command on a stream key in a specific state, with a specially crafted COUNT argument may cause an integer overflow, a subsequent heap overflow, and potentially lead to remote code execution. The problem affects Redis versions 7.0.0 or newer.
-
Shaya Potter authored
When RM_Call was used with `M` (reject OOM), `W` (reject writes), as well as `S` (rejecting stale or write commands in "Script mode"), it would have only checked the command flags, but not the declared script flag in case it's a command that runs a script. Refactoring: extracts out similar code in server.c's processCommand to be usable in RM_Call as well. (cherry picked from commit bed6d759)
-
Shay Fadida authored
When using `INFO ALL <section>`, when `section` is a specific module section. Redis will not print the additional section(s). The fix in this case, will search the modules info sections if the user provided additional sections to `ALL`. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit eedb8b17)
-
sundb authored
This PR mainly deals with 2 crashes introduced in #9357, and fix the QUICKLIST-PACKED-THRESHOLD mess in external test mode. 1. Fix crash due to deleting an entry from a compress quicklistNode When inserting a large element, we need to create a new quicklistNode first, and then delete its previous element, if the node where the deleted element is located is compressed, it will cause a crash. Now add `dont_compress` to quicklistNode, if we want to use a quicklistNode after some operation, we can use this flag like following: ```c node->dont_compress = 1; /* Prevent to be compressed */ some_operation(node); /* This operation might try to compress this node */ some_other_operation(node); /* We can use this node without decompress it */ node->dont_compress = 0; /* Re-able compression */ quicklistCompressNode(node); ``` Perhaps in the future, we could just disable the current entry from being compressed during the iterator loop, but that would require more work. 2. Fix crash due to wrongly split quicklist before #9357, the offset param of _quicklistSplitNode() will not negative. For now, when offset is negative, the split extent will be wrong. following example: ```c int orig_start = after ? offset + 1 : 0; int orig_extent = after ? -1 : offset; int new_start = after ? 0 : offset; int new_extent = after ? offset + 1 : -1; # offset: -2, after: 1, node->count: 2 # current wrong range: [-1,-1] [0,-1] # correct range: [1,-1] [0, 1] ``` Because only `_quicklistInsert()` splits the quicklistNode and only `quicklistInsertAfter()`, `quicklistInsertBefore()` call _quicklistInsert(), so `quicklistReplaceEntry()` and `listTypeInsert()` might occur this crash. But the iterator of `listTypeInsert()` is alway from head to tail(iter->offset is always positive), so it is not affected. The final conclusion is this crash only occur when we insert a large element with negative index into a list, that affects `LSET` command and `RM_ListSet` module api. 3. In external test mode, we need to restore quicklist packed threshold after when the end of test. 4. Show `node->count` in quicklistRepr(). 5. Add new tcl proc `config_get_set` to support restoring config in tests. (cherry picked from commit 13d25dd9)
-
Oran Agra authored
Redis 7.0 has #9890 which added an assertion when the propagation queue was not flushed and we got to beforeSleep. But it turns out that when processCommands calls getNodeByQuery and decides to reject the command, it can lead to a key that was lazy expired and is deleted without later flushing the propagation queue. This change prevents lazy expiry from deleting the key at this stage (not as part of a command being processed in `call`) (cherry picked from commit c789fb0a)
-
DarrenJiang13 authored
Fix bug with scripts ignoring client tracking NOLOOP and send an invalidation message anyway. (cherry picked from commit 44859a41)
-
Huang Zhw authored
`bitfield` with `get` may not be readonly. ``` 127.0.0.1:6384> acl setuser hello on nopass %R~* +@all OK 127.0.0.1:6384> auth hello 1 OK 127.0.0.1:6384> bitfield hello set i8 0 1 (error) NOPERM this user has no permissions to access one of the keys used as arguments 127.0.0.1:6384> bitfield hello set i8 0 1 get i8 0 1) (integer) 0 2) (integer) 1 ``` Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit ec5034a2)
-
Meir Shpilraien (Spielrein) authored
Fix #11030, use lua_rawget to avoid triggering metatables. #11030 shows how return `_G` from the Lua script (either function or eval), cause the Lua interpreter to Panic and the Redis processes to exit with error code 1. Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable such that the metatable raises an error. The following example demonstrate the issue: ``` 127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0 Error: Server closed the connection ``` ``` PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo') ``` The Lua panic happened because when returning the result to the client, Redis needs to introspect the returning table and transform the table into a resp. In order to scan the table, Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error. This code is not running inside `pcall` (Lua protected call), so raising an error causes the Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1. Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable that raises error when trying to fetch a none existing key. ### Solution Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget` that simply return the value from the table without triggering any metatable logic. This is promised not to raise and error. The downside of this solution is that it might be considered as breaking change, if someone rely on metatable in the returned value. An alternative solution is to wrap this entire logic with `pcall` (Lua protected call), this alternative require a much bigger refactoring. ### Back Porting The same fix will work on older versions as well (6.2, 6.0). Notice that on those version, the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done from inside the script itself. ### Tests Tests was added the verify the fix (cherry picked from commit 020e046b)
-
- 18 Jul, 2022 1 commit
-
-
Oran Agra authored
The temporary array for deleted entries reply of XAUTOCLAIM was insufficient, but also in fact the COUNT argument should be used to control the size of the reply, so instead of terminating the loop by only counting the claimed entries, we'll count deleted entries as well. Fix #10968 Addresses CVE-2022-31144 (cherry picked from commit 2825b605)
-
- 11 Jul, 2022 1 commit
-
-
Binbin authored
In #9389, we add a new `cluster-port` config and make cluster bus port configurable, and currently redis-cli --cluster create/add-node doesn't support with a configurable `cluster-port` instance. Because redis-cli uses the old way (port + 10000) to send the `CLUSTER MEET` command. Now we add this support on redis-cli `--cluster`, note we don't need to explicitly pass in the `cluster-port` parameter, we can get the real `cluster-port` of the node in `clusterManagerNodeLoadInfo`, so the `--cluster create` and `--cluster add-node` interfaces have not changed. We will use the `cluster-port` when we are doing `CLUSTER MEET`, also note that `CLUSTER MEET` bus-port parameter was added in 4.0, so if the bus_port (the one in redis-cli) is 0, or equal (port + 10000), we just call `CLUSTER MEET` with 2 arguments, using the old form. Co-authored-by:
Madelyn Olson <34459052+madolson@users.noreply.github.com>
-
- 04 Jul, 2022 1 commit
-
-
Harkrishn Patro authored
Account sharded pubsub channels memory consumption in client memory usage computation to accurately evict client based on the set threshold for `maxmemory-clients`.
-
- 03 Jul, 2022 1 commit
-
-
Yossi Gottlieb authored
Before this commit, TLS tests on Ubuntu 22.04 would fail as dropped connections result with an ECONNABORTED error thrown instead of an empty read.
-
- 28 Jun, 2022 1 commit
-
-
jonnyomerredis authored
When calling CLIENT INFO/LIST, and in various debug prints, Redis is printing the number of pubsub channels / patterns the client is subscribed to. With the addition of sharded pubsub, it would be useful to print the number of keychannels the client is subscribed to as well.
-
- 26 Jun, 2022 3 commits
-
-
RinChanNOW! authored
Since the ranges of `unsigned long long` and `long long` are different, we cannot read an `unsigned long long` integer from a `RedisModuleString` by `RedisModule_StringToLongLong` . So I added two new Redis Module APIs to support the conversion between these two types: * `RedisModule_StringToULongLong` * `RedisModule_CreateStringFromULongLong` Signed-off-by:
RinChanNOWWW <hzy427@gmail.com>
-
Binbin authored
This commit has two topics. ## Passing config name and value in the same arg In #10660 (Redis 7.0.1), when we supported the config values that can start with `--` prefix (one of the two topics of that PR), we broke another pattern: `redis-server redis.config "name value"`, passing both config name and it's value in the same arg, see #10865 This wasn't a intended change (i.e we didn't realize this pattern used to work). Although this is a wrong usage, we still like to fix it. Now we support something like: ``` src/redis-server redis.conf "--maxmemory '700mb'" "--maxmemory-policy volatile-lru" --proc-title-template --my--title--template --loglevel verbose ``` ## Changes around --save Also in this PR, we undo the breaking change we made in #10660 on purpose. 1. `redis-server redis.conf --save --loglevel verbose` (missing `save` argument before anotehr argument). In 7.0.1, it was throwing an wrong arg error. Now it will work and reset the save, similar to how it used to be in 7.0.0 and 6.2.x. 3. `redis-server redis.conf --loglevel verbose --save` (missing `save` argument as last argument). In 6.2, it did not reset the save, which was a bug (inconsistent with the previous bullet). Now we will make it work and reset the save as well (a bug fix).
-
Viktor Söderqvist authored
Adding Module APIs to let the module read and set the client name of an arbitrary connection.
-
- 21 Jun, 2022 1 commit
-
-
Meir Shpilraien (Spielrein) authored
The PR fixes 2 issues: ### RM_Call crash on script mode `RM_Call` can potentially be called from a background thread where `server.current_client` are not set. In such case we get a crash on `NULL` dereference. The fix is to check first if `server.current_client` is `NULL`, if it does we should verify disc errors and readonly replica as we do to any normal clients (no masters nor AOF). ### RM_Call block OOM commands when not needed Again `RM_Call` can be executed on a background thread using a `ThreadSafeCtx`. In such case `server.pre_command_oom_state` can be irrelevant and should not be considered when check OOM state. This cause OOM commands to be blocked when not necessarily needed. In such case, check the actual used memory (and not the cached value). Notice that in order to know if the cached value can be used, we check that the ctx that was used on the `RM_Call` is a ThreadSafeCtx. Module writer can potentially abuse the API and use ThreadSafeCtx on the main thread. We consider this as a API miss used.
-
- 14 Jun, 2022 3 commits
-
-
Oran Agra authored
when we know the size of the zset we're gonna store in advance, we can check if it's greater than the listpack encoding threshold, in which case we can create a skiplist from the get go, and avoid converting the listpack to skiplist later after it was already populated.
-
Oran Agra authored
If a script made a modification and then was interrupted for taking too long. there's a chance redis will detect that a replica dropped and would like to reject write commands with NOREPLICAS due to insufficient good replicas. returning an error on a command in this case breaks the script atomicity. The same could in theory happen with READONLY, MISCONF, but i don't think these state changes can happen during script execution.
-
Oran Agra authored
I noticed that scripting.tcl uses INFO from within a script and thought it's an overkill and concluded it's nicer to use another CMD_STALE command, decided to use ECHO, and then noticed it's not at all allowed in stale mode. probably overlooked at #6843
-
- 12 Jun, 2022 1 commit
-
-
Binbin authored
The SET and BITFIELD command were added `get_keys_function` in #10148, causing them to be wrongly marked movablekeys in `populateCommandMovableKeys`. This was an unintended side effect introduced in #10148 (7.0 RC1) which could cause some clients an extra round trip for these commands in cluster mode. Since we define movablekeys as a way to determine if the legacy range [first, last, step] doesn't find all keys, then we need a completely different approach. The right approach should be to check if the legacy range covers all key-specs, and if none of the key-specs have the INCOMPLETE flag. This way, we don't need to look at getkeys_proc of VARIABLE_FLAG at all. Probably with the exception of modules, who may still not be using key-specs. In this PR, we removed `populateCommandMovableKeys` and put its logic in `populateCommandLegacyRangeSpec`. In order to properly serve both old and new modules, we must probably keep relying CMD_MODULE_GETKEYS, but do that only for modules that don't declare key-specs. For ones that do, we need to take the same approach we take with native redis commands. This approach was proposed by Oran. Fixes #10833 Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 02 Jun, 2022 1 commit
-
-
zhaozhao.zz authored
Redis 7 adds some new alias config like `hash-max-listpack-entries` alias `hash-max-ziplist-entries`. If a config file contains both real name and alias like this: ``` hash-max-listpack-entries 20 hash-max-ziplist-entries 20 ``` after set `hash-max-listpack-entries` to 100 and `config rewrite`, the config file becomes to: ``` hash-max-listpack-entries 100 hash-max-ziplist-entries 20 ``` we can see that the alias config is not modified, and users will get wrong config after restart. 6.0 and 6.2 doesn't have this bug, since they only have the `slave` word alias. Co-authored-by:
Oran Agra <oran@redislabs.com>
-