- 11 Dec, 2020 1 commit
-
-
Yossi Gottlieb authored
This adds a new `tls-client-cert-file` and `tls-client-key-file` configuration directives which make it possible to use different certificates for the TLS-server and TLS-client functions of Redis. This is an optional directive. If it is not specified the `tls-cert-file` and `tls-key-file` directives are used for TLS client functions as well. Also, `utils/gen-test-certs.sh` now creates additional server-only and client-only certs and will skip intensive operations if target files already exist.
-
- 09 Dec, 2020 1 commit
-
-
Yossi Gottlieb authored
This adds a copy callback for module data types, in order to make modules compatible with the new COPY command. The callback is optional and COPY will fail for keys with data types that do not implement it.
-
- 08 Dec, 2020 4 commits
-
-
Oran Agra authored
Module blocked clients cache the response in a temporary client, the reply list in this client would be affected by the recent fix in #7202, but when the reply is later copied into the real client, it would have bypassed all the checks for output buffer limit, which would have resulted in both: responding with a partial response to the client, and also not disconnecting it at all.
-
Oran Agra authored
c4fdf09c added a test that now fails with valgrind it fails for two resons: 1) the test samples the used memory and then limits the maxmemory to that value, but it turns out this is not atomic and on slow machines the background cron process that clean out old query buffers reduces the memory so that the setting doesn't cause eviction. 2) the dbsize was tested late, after reading some invalidation messages by that time more and more keys got evicted, partially draining the db. this is not the focus of this fix (still a known limitation)
-
Wang Yuan authored
had some unused variables. now some are used to assert that they match, others were useless.
-
Itamar Haber authored
-
- 07 Dec, 2020 2 commits
-
-
guybe7 authored
when the same consumer re-claim an entry that it already has, there's no need to remove-and-insert if it's the same rax. we do need to update the idle time though. this commit only improves efficiency (doesn't change behavior).
-
Yossi Gottlieb authored
* Add CLIENT INFO subcommand. The output is identical to CLIENT LIST but provides a single line for the current client only. * Add CLIENT LIST ID [id...]. Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 06 Dec, 2020 5 commits
-
-
Oran Agra authored
The test creates keys with various encodings, DUMP them, corrupt the payload and RESTORES it. It utilizes the recently added use-exit-on-panic config to distinguish between asserts and segfaults. If the restore succeeds, it runs random commands on the key to attempt to trigger a crash. It runs in two modes, one with deep sanitation enabled and one without. In the first one we don't expect any assertions or segfaults, in the second one we expect assertions, but no segfaults. We also check for leaks and invalid reads using valgrind, and if we find them we print the commands that lead to that issue. Changes in the code (other than the test): - Replace a few NPD (null pointer deference) flows and division by zero with an assertion, so that it doesn't fail the test. (since we set the server to use `exit` rather than `abort` on assertion). - Fix quite a lot of flows in rdb.c that could have lead to memory leaks in RESTORE command (since it now responds with an error rather than panic) - Add a DEBUG flag for SET-SKIP-CHECKSUM-VALIDATION so that the test don't need to bother with faking a valid checksum - Remove a pile of code in serverLogObjectDebugInfo which is actually unsafe to run in the crash report (see comments in the code) - fix a missing boundary check in lzf_decompress test suite infra improvements: - be able to run valgrind checks before the process terminates - rotate log files when restarting servers
-
Oran Agra authored
- improve stream rdb encoding test to include more types of stream metadata - add test to cover various ziplist encoding entries (although it does look like the stress test above it is able to find some too - add another test for ziplist encoding for hash with full sanitization - add similar ziplist encoding tests for list
-
Oran Agra authored
When client tracking is enabled signalModifiedKey can increase memory usage, this can cause the loop in performEvictions to keep running since it was measuring the memory usage impact of signalModifiedKey. The section that measures the memory impact of the eviction should be just on dbDelete, excluding keyspace notification, client tracking, and propagation to AOF and replicas. This resolves part of the problem described in #8069 p.s. fix took 1 minute, test took about 3 hours to write.
-
guybe7 authored
One way this was happening is when a module issued an RM_Call which would inject MULTI. If the module command that does that was itself issued by something else that already did added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI. In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't the right indication that we need to propagate MULTI or not, because on a nested calls (possibly a module action called by a keyspace event of another module action), these flags aren't retained / reflected. instead there's now a global propagate_in_transaction flag for that. in addition to that, we now have a global in_eval and in_exec flags, to serve the flags of RM_GetContextFlags, since their dependence on the current client is wrong for the same reasons mentioned above.
-
Wang Yuan authored
As we know, redis may reject user's requests or evict some keys if used memory is over maxmemory. Dictionaries expanding may make things worse, some big dictionaries, such as main db and expires dict, may eat huge memory at once for allocating a new big hash table and be far more than maxmemory after expanding. There are related issues: #4213 #4583 More details, when expand dict in redis, we will allocate a new big ht[1] that generally is double of ht[0], The size of ht[1] will be very big if ht[0] already is big. For db dict, if we have more than 64 million keys, we need to cost 1GB for ht[1] when dict expands. If the sum of used memory and new hash table of dict needed exceeds maxmemory, we shouldn't allow the dict to expand. Because, if we enable keys eviction, we still couldn't add much more keys after eviction and rehashing, what's worse, redis will keep less keys when redis only remains a little memory for storing new hash table instead of users' data. Moreover users can't write data in redis if disable keys eviction. What this commit changed ? Add a new member function expandAllowed for dict type, it provide a way for caller to allow expand or not. We expose two parameters for this function: more memory needed for expanding and dict current load factor, users can implement a function to make a decision by them. For main db dict and expires dict type, these dictionaries may be very big and cost huge memory for expanding, so we implement a judgement function: we can stop dict to expand provisionally if used memory will be over maxmemory after dict expands, but to guarantee the performance of redis, we still allow dict to expand if dict load factor exceeds the safe load factor. Add test cases to verify we don't allow main db to expand when left memory is not enough, so that avoid keys eviction. Other changes: For new hash table size when expand. Before this commit, the size is that double used of dict and later _dictNextPower. Actually we aim to control a dict load factor between 0.5 and 1.0. Now we replace *2 with +1, since the first check is that used >= size, the outcome of before will usually be the same as _dictNextPower(used+1). The only case where it'll differ is when dict_can_resize is false during fork, so that later the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e. _dictNextPower(1025*2) will return 4096). Fix rehash test cases due to changing algorithm of new hash table size when expand.
-
- 03 Dec, 2020 1 commit
-
-
Itamar Haber authored
Adds the ability to use exclusive (open) start and end query intervals in XRANGE and XREVRANGE queries. Fixes #6562
-
- 02 Dec, 2020 1 commit
-
-
Itamar Haber authored
-
- 01 Dec, 2020 3 commits
-
-
Madelyn Olson authored
* Fixed SET GET executing on wrong type Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
sundb authored
SELECT used to read the index into a `long` variable, and then pass it to a function that takes an `int`, possibly causing an overflow before the range check. Now all these commands use better and cleaner range check, and that also results in a slight change of the error response in case of an invalid database index. SELECT: in the past it would have returned either `-ERR invalid DB index` (if not a number), or `-ERR DB index is out of range` (if not between 1..16 or alike). now it'll return either `-ERR value is out of range` (if not a number), or `-ERR value is out of range, value must between -2147483648 and 2147483647` (if not in the range for an int), or `-ERR DB index is out of range` (if not between 0..16 or alike) MOVE: in the past it would only fail with `-ERR index out of range` no matter the reason. now return the same errors as the new ones for SELECT mentioned above. (i.e. unlike for SELECT even for a value like 17 we changed the error message) COPY: doesn't really matter how it behaved in the past (new command), new behavior is like the above two.
-
Itamar Haber authored
Fixes #7923. This PR appropriates the special `&` symbol (because `@` and `*` are taken), followed by a literal value or pattern for describing the Pub/Sub patterns that an ACL user can interact with. It is similar to the existing key patterns mechanism in function (additive) and implementation (copy-pasta). It also adds the allchannels and resetchannels ACL keywords, naturally. The default user is given allchannels permissions, whereas new users get whatever is defined by the acl-pubsub-default configuration directive. For backward compatibility in 6.2, the default of this directive is allchannels but this is likely to be changed to resetchannels in the next major version for stronger default security settings. Unless allchannels is set for the user, channel access permissions are checked as follows : * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the argumentative channel name(s) exists for the user. * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument literally exist(s) in the user's list. Such failures are logged to the ACL log. Runtime changes to channel permissions for a user with existing subscribing clients cause said clients to disconnect unless the new permissions permit the connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be disconnected. Notes/questions: * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons for touching them.
-
- 29 Nov, 2020 1 commit
-
-
guybe7 authored
Used to filter stream pending entries by their idle-time, useful for XCLAIMing entries that have not been processed for some time
-
- 23 Nov, 2020 1 commit
-
-
Itamar Haber authored
-
- 18 Nov, 2020 1 commit
-
-
guybe7 authored
The bug was introduced by #5021 which only attempted avoid EXIST on an already expired key from returning 1 on a replica. Before that commit, dbExists was used instead of lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU) Other than that, this commit fixes OBJECT to also come empty handed on expired keys in replica. And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from the key regardless of it's expired state)
-
- 17 Nov, 2020 3 commits
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
thomaston authored
ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
swamp0407 authored
Syntax: COPY <key> <new-key> [DB <dest-db>] [REPLACE] No support for module keys yet. Co-authored-by: tmgauss Co-authored-by:
Itamar Haber <itamar@redislabs.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 16 Nov, 2020 1 commit
-
-
chenyangyang authored
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort` and `unlink`. the `free_effort` callback indicates the effort required to free a module memory. Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and may soon be freed by a background thread. Add `lazyfreed_objects` info field, which represents the number of objects that have been lazyfreed since redis was started. Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
-
- 15 Nov, 2020 1 commit
-
-
Felipe Machado authored
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE - Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 11 Nov, 2020 1 commit
-
-
Yossi Gottlieb authored
-
- 10 Nov, 2020 1 commit
-
-
Madelyn Olson authored
* Rewritten commands are logged as their original command Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 09 Nov, 2020 1 commit
-
-
nitaicaro authored
Test support for the new map, null and push message types. Map objects are parsed as a list of lists of key value pairs. for instance: user => john password => 123 will be parsed to the following TCL list: {{user john} {password 123}} Also added the following tests: Redirection still works with RESP3 Able to use a RESP3 client as a redirection client No duplicate invalidation messages when turning BCAST mode on after normal tracking Server is able to evacuate enough keys when num of keys surpasses limit by more than defined initial effort Different clients using different protocols can track the same key OPTOUT tests OPTIN tests Clients can redirect to the same connection tracking-redir-broken test HELLO 3 checks Invalidation messages still work when using RESP3, with and without redirection Switching to RESP3 doesn't disturb previous tracked keys Tracking info is correct Flushall and flushdb produce invalidation messages These tests achieve 100% line coverage for tracking.c using lcov.
-
- 08 Nov, 2020 1 commit
-
-
sundb authored
-
- 05 Nov, 2020 1 commit
-
-
Yossi Gottlieb authored
Perform full reset of all client connection states, is if the client was disconnected and re-connected. This affects: * MULTI state * Watched keys * MONITOR mode * Pub/Sub subscription * ACL/Authenticated state * Client tracking state * Cluster read-only/asking state * RESP version (reset to 2) * Selected database * CLIENT REPLY state The response is +RESET to make it easily distinguishable from other responses. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 04 Nov, 2020 1 commit
-
-
Yossi Gottlieb authored
Not disabling save, slower systems begun background save that did not complete in time, resulting with SAVE failing with "ERR Background save already in progress".
-
- 03 Nov, 2020 1 commit
-
- 28 Oct, 2020 1 commit
-
-
Madelyn Olson authored
-
- 22 Oct, 2020 2 commits
-
-
Qu Chen authored
This wrong behavior was backed by a test, and also documentation, and dates back to 2010. But it makes no sense to anyone involved so it was decided to change that. Note that 20eeddfb (invalidate watch on expire on access) was released in 6.0 RC2 and 2d1968f8 released in in 6.0.0 GA (invalidate watch when key is evicted). both of which do similar changes.
-
Yossi Gottlieb authored
Useful for running tests on systems which may be way slower than usual.
-
- 18 Oct, 2020 1 commit
-
-
Wen Hui authored
introduces a NOMKSTREAM option for xadd command, this would be useful for some use cases when we do not want to create new stream by default: XADD key [MAXLEN [~|=] <count>] [NOMKSTREAM] <ID or *> [field value] [field value]
-
- 11 Oct, 2020 3 commits
-
-
Meir Shpilraien (Spielrein) authored
* Introduce a new API's: RM_GetContextFlagsAll, and RM_GetKeyspaceNotificationFlagsAll that will return the full flags mask of each feature. The module writer can check base on this value if the Flags he needs are supported or not. * For each flag, introduce a new value on redismodule.h, this value represents the LAST value and should be there as a reminder to update it when a new value is added, also it will be used in the code to calculate the full flags mask (assuming flags are incrementally increasing). In addition, stated that the module writer should not use the LAST flag directly and he should use the GetFlagAll API's. * Introduce a new API: RM_IsSubEventSupported, that returns for a given event and subevent, whether or not the subevent supported. * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether or not a function API is supported by comparing it to NULL. * Introduce a new API: int RM_GetServerVersion();, that will return the current Redis version in the format 0x00MMmmpp; e.g. 0x00060008; * Changed unstable version from 999.999.999 to 255.255.255 Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
Yossi Gottlieb authored
This API function makes it possible to retrieve the X.509 certificate used by clients to authenticate TLS connections.
-
Yossi Gottlieb authored
The main motivation here is to provide a way for modules to create a single, global context that can be used for logging. Currently, it is possible to obtain a thread-safe context that is not attached to any blocked client by using `RM_GetThreadSafeContext`. However, the attached context is not linked to the module identity so log messages produced are not tagged with the module name. Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it doesn't accept the current context as an argument there's no way to do that in a backwards compatible manner.
-