- 25 Dec, 2020 1 commit
-
-
Itamar Haber authored
Adds: `L/RPOP <key> [count]` Implements no. 2 of the following strategies: 1. Loop on listTypePop - this would result in multiple calls for memory freeing and allocating (see https://github.com/redis/redis/pull/8179/commits/769167a079b0e110d28e4a8099dce1ecd45682b5) 2. Iterate the range to build the reply, then call quickListDelRange - this requires two iterations and **is the current choice** 3. Refactor quicklist to have a pop variant of quickListDelRange - probably optimal but more complex Also: * There's a historical check for NULL after calling listTypePop that was converted to an assert. * This refactors common logic shared between LRANGE and the new form of LPOP/RPOP into addListRangeReply (adds test for b/w compat) * Consequently, it may have made sense to have `LRANGE l -1 -2` and `LRANGE l 9 0` be legit and return a reverse reply. Due to historical reasons that would be, however, a breaking change. * Added minimal comments to existing commands to adhere to the style, make core dev life easier and get commit karma, naturally.
-
- 24 Dec, 2020 2 commits
-
-
Oran Agra authored
The commit deals with the syncWithMaster and the ugly state machine in it. It attempts to clean it a bit, but more importantly it uses pipeline for part of the work (rather than 7 round trips, we now have 4). i.e. the connect and PING are separate, then AUTH + 3 REPLCONF in one pipeline, and finally the PSYNC (must be separate since the master has to have an empty output buffer).
-
Madelyn Olson authored
Cleanup key tracking documentation, always cleanup the tracking table, and free the tracking table in an async manner when applicable.
-
- 22 Dec, 2020 2 commits
-
-
Oran Agra authored
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
-
sundb authored
* Fix some redundancy use of semicolon in do-while macros
-
- 17 Dec, 2020 1 commit
-
-
Qu Chen authored
* Handle binary safe string for REQUIREPASS and MASTERAUTH directives.
-
- 13 Dec, 2020 2 commits
-
-
Wang Yuan authored
-
Yossi Gottlieb authored
Add a new set of defrag functions that take a defrag context and allow defragmenting memory blocks and RedisModuleStrings. Modules can register a defrag callback which will be invoked when the defrag process handles globals. Modules with custom data types can also register a datatype-specific defrag callback which is invoked for keys that require defragmentation. The callback and associated functions support both one-step and multi-step options, depending on the complexity of the key as exposed by the free_effort callback.
-
- 12 Dec, 2020 1 commit
-
-
杨博东 authored
Add commands to query geospatial data with bounding box. Two new commands that replace the existing 4 GEORADIUS* commands. GEOSEARCH key [FROMMEMBER member] [FROMLOC long lat] [BYRADIUS radius unit] [BYBOX width height unit] [WITHCORD] [WITHDIST] [WITHASH] [COUNT count] [ASC|DESC] GEOSEARCHSTORE dest_key src_key [FROMMEMBER member] [FROMLOC long lat] [BYRADIUS radius unit] [BYBOX width height unit] [WITHCORD] [WITHDIST] [WITHASH] [COUNT count] [ASC|DESC] [STOREDIST] - Add two types of CIRCULAR_TYPE and RECTANGLE_TYPE to achieve different searches - Judge whether the point is within the rectangle, refer to: geohashGetDistanceIfInRectangle
-
- 11 Dec, 2020 1 commit
-
-
Yossi Gottlieb authored
This adds a new `tls-client-cert-file` and `tls-client-key-file` configuration directives which make it possible to use different certificates for the TLS-server and TLS-client functions of Redis. This is an optional directive. If it is not specified the `tls-cert-file` and `tls-key-file` directives are used for TLS client functions as well. Also, `utils/gen-test-certs.sh` now creates additional server-only and client-only certs and will skip intensive operations if target files already exist.
-
- 09 Dec, 2020 1 commit
-
-
Yossi Gottlieb authored
This adds a copy callback for module data types, in order to make modules compatible with the new COPY command. The callback is optional and COPY will fail for keys with data types that do not implement it.
-
- 06 Dec, 2020 6 commits
-
-
Oran Agra authored
First, if the ziplist header is surely inside the ziplist, do fast path decoding rather than the careful one. In that case, streamline the encoding if-else chain to be executed only once, and the encoding validity tested at the end. encourage inlining likely / unlikely hints for speculative execution Assertion used _exit(1) to tell the compiler that the code after them is not reachable and get rid of warnings. But in some cases assertions are placed inside tight loops, and any piece of code in them can slow down execution (code cache and other reasons), instead using either abort() or better yet, unreachable builtin.
-
Oran Agra authored
If RESTORE passes successfully with full sanitization, we can't affort to crash later on assertion due to duplicate records in a hash when converting it form ziplist to dict. This means that when doing full sanitization, we must make sure there are no duplicate records in any of the collections.
-
Oran Agra authored
The test creates keys with various encodings, DUMP them, corrupt the payload and RESTORES it. It utilizes the recently added use-exit-on-panic config to distinguish between asserts and segfaults. If the restore succeeds, it runs random commands on the key to attempt to trigger a crash. It runs in two modes, one with deep sanitation enabled and one without. In the first one we don't expect any assertions or segfaults, in the second one we expect assertions, but no segfaults. We also check for leaks and invalid reads using valgrind, and if we find them we print the commands that lead to that issue. Changes in the code (other than the test): - Replace a few NPD (null pointer deference) flows and division by zero with an assertion, so that it doesn't fail the test. (since we set the server to use `exit` rather than `abort` on assertion). - Fix quite a lot of flows in rdb.c that could have lead to memory leaks in RESTORE command (since it now responds with an error rather than panic) - Add a DEBUG flag for SET-SKIP-CHECKSUM-VALIDATION so that the test don't need to bother with faking a valid checksum - Remove a pile of code in serverLogObjectDebugInfo which is actually unsafe to run in the crash report (see comments in the code) - fix a missing boundary check in lzf_decompress test suite infra improvements: - be able to run valgrind checks before the process terminates - rotate log files when restarting servers
-
Oran Agra authored
When loading an encoded payload we will at least do a shallow validation to check that the size that's encoded in the payload matches the size of the allocation. This let's us later use this encoded size to make sure the various offsets inside encoded payload don't reach outside the allocation, if they do, we'll assert/panic, but at least we won't segfault or smear memory. We can also do 'deep' validation which runs on all the records of the encoded payload and validates that they don't contain invalid offsets. This lets us detect corruptions early and reject a RESTORE command rather than accepting it and asserting (crashing) later when accessing that payload via some command. configuration: - adding ACL flag skip-sanitize-payload - adding config sanitize-dump-payload [yes/no/clients] For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't being slowed down by these sanitation, so i'm setting the default value to `no`, but later on it should be set to `clients` by default. changes: - changing rdbReportError not to `exit` in RESTORE command - adding a new stat to be able to later check if cluster MIGRATE isn't being slowed down by sanitation.
-
guybe7 authored
One way this was happening is when a module issued an RM_Call which would inject MULTI. If the module command that does that was itself issued by something else that already did added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI. In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't the right indication that we need to propagate MULTI or not, because on a nested calls (possibly a module action called by a keyspace event of another module action), these flags aren't retained / reflected. instead there's now a global propagate_in_transaction flag for that. in addition to that, we now have a global in_eval and in_exec flags, to serve the flags of RM_GetContextFlags, since their dependence on the current client is wrong for the same reasons mentioned above.
-
Wang Yuan authored
As we know, redis may reject user's requests or evict some keys if used memory is over maxmemory. Dictionaries expanding may make things worse, some big dictionaries, such as main db and expires dict, may eat huge memory at once for allocating a new big hash table and be far more than maxmemory after expanding. There are related issues: #4213 #4583 More details, when expand dict in redis, we will allocate a new big ht[1] that generally is double of ht[0], The size of ht[1] will be very big if ht[0] already is big. For db dict, if we have more than 64 million keys, we need to cost 1GB for ht[1] when dict expands. If the sum of used memory and new hash table of dict needed exceeds maxmemory, we shouldn't allow the dict to expand. Because, if we enable keys eviction, we still couldn't add much more keys after eviction and rehashing, what's worse, redis will keep less keys when redis only remains a little memory for storing new hash table instead of users' data. Moreover users can't write data in redis if disable keys eviction. What this commit changed ? Add a new member function expandAllowed for dict type, it provide a way for caller to allow expand or not. We expose two parameters for this function: more memory needed for expanding and dict current load factor, users can implement a function to make a decision by them. For main db dict and expires dict type, these dictionaries may be very big and cost huge memory for expanding, so we implement a judgement function: we can stop dict to expand provisionally if used memory will be over maxmemory after dict expands, but to guarantee the performance of redis, we still allow dict to expand if dict load factor exceeds the safe load factor. Add test cases to verify we don't allow main db to expand when left memory is not enough, so that avoid keys eviction. Other changes: For new hash table size when expand. Before this commit, the size is that double used of dict and later _dictNextPower. Actually we aim to control a dict load factor between 0.5 and 1.0. Now we replace *2 with +1, since the first check is that used >= size, the outcome of before will usually be the same as _dictNextPower(used+1). The only case where it'll differ is when dict_can_resize is false during fork, so that later the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e. _dictNextPower(1025*2) will return 4096). Fix rehash test cases due to changing algorithm of new hash table size when expand.
-
- 02 Dec, 2020 1 commit
-
-
Wang Yuan authored
Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108) When replica diskless-load type is swapdb in cluster mode, we didn't backup keys to slots map, so we will lose keys to slots map if fail to sync. Now we backup keys to slots map at first, and restore it properly when fail. This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit). Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 01 Dec, 2020 2 commits
-
-
sundb authored
SELECT used to read the index into a `long` variable, and then pass it to a function that takes an `int`, possibly causing an overflow before the range check. Now all these commands use better and cleaner range check, and that also results in a slight change of the error response in case of an invalid database index. SELECT: in the past it would have returned either `-ERR invalid DB index` (if not a number), or `-ERR DB index is out of range` (if not between 1..16 or alike). now it'll return either `-ERR value is out of range` (if not a number), or `-ERR value is out of range, value must between -2147483648 and 2147483647` (if not in the range for an int), or `-ERR DB index is out of range` (if not between 0..16 or alike) MOVE: in the past it would only fail with `-ERR index out of range` no matter the reason. now return the same errors as the new ones for SELECT mentioned above. (i.e. unlike for SELECT even for a value like 17 we changed the error message) COPY: doesn't really matter how it behaved in the past (new command), new behavior is like the above two.
-
Itamar Haber authored
Fixes #7923. This PR appropriates the special `&` symbol (because `@` and `*` are taken), followed by a literal value or pattern for describing the Pub/Sub patterns that an ACL user can interact with. It is similar to the existing key patterns mechanism in function (additive) and implementation (copy-pasta). It also adds the allchannels and resetchannels ACL keywords, naturally. The default user is given allchannels permissions, whereas new users get whatever is defined by the acl-pubsub-default configuration directive. For backward compatibility in 6.2, the default of this directive is allchannels but this is likely to be changed to resetchannels in the next major version for stronger default security settings. Unless allchannels is set for the user, channel access permissions are checked as follows : * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the argumentative channel name(s) exists for the user. * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument literally exist(s) in the user's list. Such failures are logged to the ACL log. Runtime changes to channel permissions for a user with existing subscribing clients cause said clients to disconnect unless the new permissions permit the connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be disconnected. Notes/questions: * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons for touching them.
-
- 22 Nov, 2020 1 commit
-
-
Oran Agra authored
Fix: When oom-score-adj-values is provided in the config file after oom-score-adj yes, it'll take an immediate action, before readOOMScoreAdj was acquired, resulting in an error (out of range score due to uninitialized value. delay the reaction the real call is made by main(). Since the values are clamped to -1000..1000, and they're applied as an offset from the value at startup (which may be -1000), we need to allow the offsets to reach to +2000 so that a value of +1000 is achievable in case the value at startup was -1000. Adding an option for absolute values rather than relative ones.
-
- 18 Nov, 2020 1 commit
-
-
guybe7 authored
The bug was introduced by #5021 which only attempted avoid EXIST on an already expired key from returning 1 on a replica. Before that commit, dbExists was used instead of lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU) Other than that, this commit fixes OBJECT to also come empty handed on expired keys in replica. And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from the key regardless of it's expired state)
-
- 17 Nov, 2020 2 commits
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
swamp0407 authored
Syntax: COPY <key> <new-key> [DB <dest-db>] [REPLACE] No support for module keys yet. Co-authored-by: tmgauss Co-authored-by:
Itamar Haber <itamar@redislabs.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 16 Nov, 2020 1 commit
-
-
chenyangyang authored
Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort` and `unlink`. the `free_effort` callback indicates the effort required to free a module memory. Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and may soon be freed by a background thread. Add `lazyfreed_objects` info field, which represents the number of objects that have been lazyfreed since redis was started. Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
-
- 15 Nov, 2020 1 commit
-
-
Felipe Machado authored
- Add ZDIFF and ZDIFFSTORE which work similarly to SDIFF and SDIFFSTORE - Make sure the new WITHSCORES argument that was added for ZUNION isn't considered valid for ZUNIONSTORE Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 12 Nov, 2020 1 commit
-
-
Yash Ladha authored
BLPOP when there are elements in the list works in the same way as LPOP does. Due to this they also does the same repetitive action and logic for the same is written at two different places. This is a bad code practice as the one needs the context to change the BLPOP list pop code as well when the LPOP code gets changed. Separated the generic logic from LPOP to a function that is being used by the BLPOP code as well.
-
- 10 Nov, 2020 1 commit
-
-
Madelyn Olson authored
* Rewritten commands are logged as their original command Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
- 08 Nov, 2020 1 commit
-
-
bugwz authored
use 40 zeros rather than 30 zeros to match CONFIG_RUN_ID_SIZE. this doesn't have any real implications.
-
- 05 Nov, 2020 3 commits
-
-
Wen Hui authored
* Debug Populate: Add checks for count and keysize to avoid crash * provide getRangeLongFromObjectOrReply and getPositiveLongFromObjectOrReply for range check
-
Oran Agra authored
Expose new `loading_rdb_used_mem` showing the used memory of the server that saved the RDB file we're currently using. This is useful in diskless replication when the total size of the rdb is unkown, and can be used as a rought estimation of progres. Use that new field to calculate the "user friendly" `loading_loaded_perc` and `loading_eta_seconds`. Expose `master_sync_total_bytes` and `master_sync_total_bytes` to complement on the existing `master_sync_total_bytes` (which cannot be used on its own to calculate progress). Add "user friendly" field for `master_sync_perc`
-
Yossi Gottlieb authored
Perform full reset of all client connection states, is if the client was disconnected and re-connected. This affects: * MULTI state * Watched keys * MONITOR mode * Pub/Sub subscription * ACL/Authenticated state * Client tracking state * Cluster read-only/asking state * RESP version (reset to 2) * Selected database * CLIENT REPLY state The response is +RESET to make it easily distinguishable from other responses. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 28 Oct, 2020 1 commit
-
-
yoav-steinberg authored
Useful when you want to know through which bind address the client connected to the server in case of multiple bind addresses. - Adding `laddr` field to CLIENT list showing the local (bind) address. - Adding `LADDR` option to CLIENT KILL to kill all the clients connected to a specific local address. - Refactoring to share code.
-
- 27 Oct, 2020 1 commit
-
-
zhenwei pi authored
In case redis starts and find that THP is enabled ("always"), instead of printing a log message, which might go unnoticed, redis will try to disable it (just for the redis process). Note: it looks like on self-bulit kernels THP is likely be set to "always" by default. Some discuss about THP side effect on Linux: according to http://www.antirez.com/news/84 , we can see that redis latency spikes are caused by linux kernel THP feature. I have tested on E3-2650 v3, and found that 2M huge page costs about 0.25ms to fix COW page fault. Add a new config 'disable-thp', the recommended setting is 'yes', (default) the redis tries to disable THP by prctl syscall. But users who really want THP can set it to "no" Thanks to Oran & Yossi for suggestions. Signed-off-by:
zhenwei pi <pizhenwei@bytedance.com>
-
- 19 Oct, 2020 1 commit
-
-
Wen Hui authored
This commit implements ACL for Sentinel mode, main work of this PR includes: - Update Sentinel command table in order to better support ACLs. - Fix couple of things which currently blocks the support for ACL on sentinel mode. - Provide "sentinel sentinel-user" and "sentinel sentinel-pass " configuration in order to let sentinel authenticate with a specific user in other sentinels. - requirepass is kept just for compatibility with old config files Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 18 Oct, 2020 1 commit
-
-
Tommy Joe Lund authored
-
- 13 Oct, 2020 1 commit
-
-
WuYunlong authored
-
- 11 Oct, 2020 2 commits
-
-
Yossi Gottlieb authored
Avoid using a static buffer for short key index responses, and make it caller's responsibility to stack-allocate a result type. Responses that don't fit are still allocated on the heap.
-
Uri Shachar authored
This allows supplying secret configuration (for example - masterauth) via a secure channel instead of having it in a plaintext file / command line param, while still allowing for most of the configuration to reside there. Also, remove 'special' case handling for --check-rdb which hasn't been relevant since 4.0.0.
-
- 08 Oct, 2020 1 commit
-
-
Felipe Machado authored
Adding [B]LMOVE <src> <dst> RIGHT|LEFT RIGHT|LEFT. deprecating [B]RPOPLPUSH. Note that when receiving a BRPOPLPUSH we'll still propagate an RPOPLPUSH, but on BLMOVE RIGHT LEFT we'll propagate an LMOVE improvement to existing tests - Replace "after 1000" with "wait_for_condition" when wait for clients to block/unblock. - Add a pre-existing element to target list on basic tests so that we can check if the new element was added to the correct side of the list. - check command stats on the replica to make sure the right command was replicated Co-authored-by:
Oran Agra <oran@redislabs.com>
-