- 10 Mar, 2021 2 commits
-
-
guybe7 authored
Bug 1: When a module ctx is freed moduleHandlePropagationAfterCommandCallback is called and handles propagation. We want to prevent it from propagating commands that were not replicated by the same context. Example: 1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3) 2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte 3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx. 4. moduleHandlePropagationAfterCommandCallback is called, calling alsoPropagates EXEC (Note: EXEC is still not written to socket), setting server.in_trnsaction = 0 5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now we have nested MULTI calls, which is no good) and then cmd3 We must prevent RM_Call(cmd2) from resetting server.in_transaction. REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose. Bug 2: Fix issues with nested RM_Call where some have '!' and some don't. Example: 1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!') 2. module2.bar internally calls RM_Call of INCR with '!' 3. at the end of module1.foo we call RM_ReplicateVerbatim We want the replica/AOF to see only module1.foo and not the INCR from module2.bar Introduced a global replication_allowed flag inside RM_Call to determine whether we need to replicate or not (even if '!' was specified) Other changes: Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec just for better readability
-
guybe7 authored
Have a clear separation between in and out flags Other changes: delete dead code in RM_ZsetIncrby: if zsetAdd returned error (happens only if the result of the operation is NAN or if score is NAN) we return immediately so there is no way that zsetAdd succeeded and returned NAN in the out-flags
-
- 05 Mar, 2021 1 commit
-
-
Huang Zhw authored
-
- 03 Mar, 2021 1 commit
-
-
Yossi Gottlieb authored
This solves the problem of /dev/random and /dev/urandom open file descriptors leaking to childs with some versions of OpenSSL.
-
- 02 Mar, 2021 1 commit
-
-
Yossi Gottlieb authored
Fixes #8574
-
- 01 Mar, 2021 1 commit
-
-
YaacovHazan authored
Add ability to modify port, tls-port and bind configurations by CONFIG SET command. To simplify the code and make it cleaner, a new structure added, socketFds, which contains the file descriptors array and its counter, and used for TCP, TLS and Cluster sockets file descriptors.
-
- 28 Feb, 2021 1 commit
-
-
Viktor Söderqvist authored
A single client pointer is added in the server struct. This is initialized by the first RM_Call() and reused for every subsequent RM_Call() except if it's already in use, which means that it's not used for (recursive) module calls to modules. For these, a new "fake" client is created each time. Other changes: * Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels when not needed
-
- 26 Feb, 2021 1 commit
-
-
Madelyn Olson authored
Moved additional configs to generic infrastructure.
-
- 21 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
Originally this was limited to IPv6 address length, but effectively it has been used for host names and now that Sentinel accepts that as well we need to be able to store full hostnames. Fixes #8507
-
- 17 Feb, 2021 1 commit
-
-
Harkrishn Patro authored
Remove redundant pubsub list to store the patterns.
-
- 16 Feb, 2021 1 commit
-
-
uriyage authored
* Adding current_save_keys_total and current_save_keys_processed info fields. Present in replication, BGSAVE and AOFRW. * Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress) * Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW, and module forks.
-
- 15 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
Fixes #8489
-
- 09 Feb, 2021 1 commit
-
-
Madelyn Olson authored
Moved most static strings into the shared structure
-
- 29 Jan, 2021 2 commits
-
-
filipe oliveira authored
Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491) This commit enables tracking time of the background tasks and on replies, opening the door for properly tracking commands that rely on blocking / background work via the slowlog, latency history, and commandstats. Some notes: - The time spent blocked waiting for key changes, or blocked on synchronous replication is not accounted for. - **This commit does not affect latency tracking of commands that are non-blocking or do not have background work.** ( meaning that it all stays the same with exception to `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely on background threads ). - Specifically for latency history command we've added a new event class named `command-unblocking` that will enable latency monitoring on commands that spawn background threads to do the work. - For blocking commands we're now considering the total time of a command as the time spent on call() + the time spent on replying when unblocked. - For Modules commands that rely on background threads we're now considering the total time of a command as the time spent on call (main thread) + the time spent on the background thread ( if marked within `RedisModule_MeasureTimeStart()` and `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread) To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on a module that blocks the client and sleeps on the background for a given time. - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout - check blocked command with multiple calls RedisModule_MeasureTimeStart() is tracking the total background time - check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
-
Yang Bodong authored
New commands: `HRANDFIELD [<count> [WITHVALUES]]` `ZRANDMEMBER [<count> [WITHSCORES]]` Algorithms are similar to the one in SRANDMEMBER. Both return a simple bulk response when no arguments are given, and an array otherwise. In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array. note: in all 3 commands, the only option that also provides random order is the one with negative count. Changes to SRANDMEMBER * Optimization when count is 1, we can use the more efficient algorithm of non-unique random * optimization: work with sds strings rather than robj Other changes: * zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in case the buffer is too small) * Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or case 4) and by accident used a negative count Co-authored-by:
xinluton <xinluton@qq.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 28 Jan, 2021 3 commits
-
-
Allen Farris authored
Implement FAILOVER command, which coordinates failover between the server and one of its replicas.
-
Yossi Gottlieb authored
Make it possible to customize the process title, i.e. include custom strings, immutable configuration like port, tls-port, unix socket name, etc.
-
Z. Liu authored
if option `set-proc-title' is no, then do nothing for proc title. The reason has been explained long ago, see following: We update redis to 2.8.8, then found there are some side effect when redis always change the process title. We run several slave instance on one computer, and all these salves listen on unix socket only, then ps will show: 1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0 1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0 for redis 2.6 the output of ps is like following: 1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf 1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf Later is more informational in our case. The situation is worse when we manage the config and process running state by salt. Salt check the process by running "ps | grep SIG" (for Gentoo System) to check the running state, where SIG is the string to search for when looking for the service process with ps. Previously, we define sig as "/usr/sbin/redis-server /etc/redis/a.conf". Since the ps output is identical for our case, so we have no way to check the state of specified redis instance. So, for our case, we prefer the old behavior, i.e, do not change the process title for the main redis process. Or add an option such as "set-proc-title [yes|no]" to control this behavior. Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com> Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 27 Jan, 2021 2 commits
-
-
Raghav Muddur authored
This commit introduces two new command and two options for an existing command GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp] [PXAT milliseconds-timestamp] The getexCommand() function implements extended options and variants of the GET command. Unlike GET command this command is not read-only. Only one of the options can be used at a given time. 1. PERSIST removes any TTL associated with the key. 2. EX Set expiry TTL in seconds. 3. PX Set expiry TTL in milliseconds. 4. EXAT Same like EX instead of specifying the number of seconds representing the TTL (time to live), it takes an absolute Unix timestamp 5. PXAT Same like PX instead of specifying the number of milliseconds representing the TTL (time to live), it takes an absolute Unix timestamp Command would return either the bulk string, error or nil. GETDEL <key> Would delete the key after getting. SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] [PX <milliseconds>] [EXAT <seconds-timestamp>][PXAT <milliseconds-timestamp>] Two new options added here are EXAT and PXAT Key implementation notes - `SET` with `PX/EX/EXAT/PXAT` is always translated to `PXAT` in `AOF`. When relative time is specified (`PX/EX`), replication will always use `PX`. - `setexCommand` and `psetexCommand` would no longer need translation in `feedAppendOnlyFile` as they are modified to invoke `setGenericCommand ` with appropriate flags which will take care of correct AOF translation. - `GETEX` without any optional argument behaves like `GET`. - `GETEX` command is never propagated, It is either propagated as `PEXPIRE[AT], or PERSIST`. - `GETDEL` command is propagated as `DEL` - Combined the validation for `SET` and `GETEX` arguments. - Test cases to validate AOF/Replication propagation
-
Wang Yuan authored
In some scenarios, such as remote backup, we only want to get remote redis server db snapshot. Currently, redis-cli acts as a replica and sends SYNC to redis, but redis still accumulates replication buffer in the replica client output buffer, that may result in using vast memory, or failing to transfer RDB because of client-output-buffer-limit. In this commit, we add 'replconf rdb-only 0|1', redis doesn't send incremental replication buffer to them if they send 'replconf rdb-only 1', so we can reduce used memory and improve success of getting RDB.
-
- 26 Jan, 2021 1 commit
-
-
Wen Hui authored
This commit fixes a well known and an annoying issue in Sentinel mode. Cause of this issue: Currently, Redis rewrite process works well in server mode, however in sentinel mode, the sentinel config has variant semantics for different configurations, in example configuration https://github.com/redis/redis/blob/unstable/sentinel.conf, we put comments on these. However the rewrite process only treat the sentinel config as a single option. During rewrite process, it will mess up with the lines and comments. Approaches: In order to solve this issue, we need to differentiate different subconfig options in sentinel separately, for example, sentinel monitor <master-name> <ip> <redis-port> <quorum> we can treat it as sentinel monitor option, instead of the sentinel option. This commit also fixes the dependency issue when putting configurations in sentinel.conf. For example before this commit,we must put `sentinel monitor <master-name> <ip> <redis-port> <quorum>` before `sentinel auth-pass <master-name> <password>` for a single master, otherwise the server cannot start and will return error. This commit fixes this issue, as long as the monitoring master was configured, no matter the sequence is, the sentinel can start and run properly.
-
- 21 Jan, 2021 1 commit
-
-
Yossi Gottlieb authored
-
- 20 Jan, 2021 1 commit
-
-
Yossi Gottlieb authored
Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one. The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly. Fixes #8369
-
- 15 Jan, 2021 1 commit
-
-
Yang Bodong authored
Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258) * Adds ASYNC and SYNC arguments to SCRIPT FLUSH * Adds SYNC argument to FLUSHDB and FLUSHALL * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH. the new behavior is as follows: * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the value of lazyfree-lazy-user-flush. * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner. * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
-
- 08 Jan, 2021 3 commits
-
-
Oran Agra authored
- the last COW report wasn't always read from the pipe (receiveLastChildInfo wasn't used) - but in fact, there's no reason we won't always try to drain that pipe so i'm unifying receiveLastChildInfo with receiveChildInfo - adjust threshold of the COW test when run in accurate mode - add some prints in case this test fails again - fix indentation, page size, and PID! in MacOS proc info p.s. it seems that pri_pages_dirtied is always 0
-
Madelyn Olson authored
Throw an error if there are conflicting bcast tracking prefixes.
-
Madelyn Olson authored
Implementation of client pause WRITE and client unpause
-
- 07 Jan, 2021 4 commits
-
-
George Prekas authored
Older arm64 Linux kernels have a bug that could lead to data corruption during background save under the following scenario: 1) jemalloc uses MADV_FREE on a page, 2) jemalloc reuses and writes the page, 3) Redis forks the background save process, and 4) Linux performs page reclamation. Under these conditions, Linux will reclaim the page wrongfully and the background save process will read zeros when it tries to read the page. The bug has been fixed in Linux with commit: ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is preserved across pte_wrprotect()") This Commit adds an ignore-warnings config, when not found, redis will print a warning and exit on startup (default behavior). Co-authored-by:
Oran Agra <oran@redislabs.com>
-
YaacovHazan authored
Add INFO field, rdb_active_cow_size, to report COW of a live fork child while it's active. - once in 1024 keys check the time, and if there's more than one second since the last report send a report to the parent via the pipe. - refactor the child_info_data struct, it's an implementation detail that shouldn't be in the server struct, and not used to communicate data between caller and callee - remove the magic value from that struct (not sure what it was good for), and instead add handling of short reads. - add another value to the structure, cow_type, to indicate if the report is for the new rdb_active_cow_size field, or it's the last report of a successful operation - add new Module API to report the active COW - add more asserts variants to test.tcl
-
YaacovHazan authored
This is a refactory commit, isn't suppose to have any actual impact. it does the following: - keep just one server struct fork child pid variable instead of 3 - have one server struct variable indicating the purpose of the current fork child. - redisFork is now responsible of updating the server struct with the pid, which means it can be the one that calls updateDictResizePolicy - move child info pipe handling into redisFork instead of having them repeated outside - there are two classes of fork purposes, mutually exclusive group (AOF, RDB, Module), and one that can create several forks to coexist in parallel (LDB, but maybe Modules some day too, Module API allows for that). - minor fix to killRDBChild: unlike killAppendOnlyChild and TerminateModuleForkChild, the killRDBChild doesn't clear the pid variable or call wait4, so checkChildrenDone does the cleanup for it. This commit removes the explicit calls to rdbRemoveTempFile, closeChildInfoPipe, updateDictResizePolicy, which didn't do any harm, but where unnecessary.
-
Jonah H. Harris authored
Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX]. Syntax for the new ZRANGESTORE command: ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count] New syntax for ZRANGE: ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count] Old syntax for ZRANGE: ZRANGE [WITHSCORES] Other ZRANGE commands remain unchanged. The implementation uses common code for all of these, by utilizing a consumer interface that in one command response to the client, and in the other command stores a zset key. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 06 Jan, 2021 1 commit
-
-
guybe7 authored
New command: XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT <count>] [JUSTID] The purpose is to claim entries from a stale consumer without the usual XPENDING+XCLAIM combo which takes two round trips. The syntax for XAUTOCLAIM is similar to scan: A cursor is returned (streamID) by each call and should be used as start for the next call. 0-0 means the scan is complete. This PR extends the deferred reply mechanism for any bulk string (not just counts) This PR carries some unrelated test code changes: - Renames the term "client" into "consumer" in the stream-cgroups test - And also changes DEBUG SLEEP into "after" Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 04 Jan, 2021 2 commits
-
-
Yang Bodong authored
This PR not only fixes the problem that swapdb does not make the transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients. FLUSHDB was changed to first iterate on all watched keys, and then on the clients watching each key. Instead of iterating though all clients, and for each iterate on watched keys. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Meir Shpilraien (Spielrein) authored
If AOF file contains a long Lua script that timed out, then the `evalCommand` calls `blockingOperationEnds` which sets `server.blocked_last_cron` to 0. later on, the AOF `whileBlockedCron` function asserts that this value is not 0. The fix allows nesting call to `blockingOperationStarts` and `blockingOperationEnds`. The issue was first introduce in this commit: 9ef8d2f6 (Redis 6.2 RC1)
-
- 31 Dec, 2020 1 commit
-
-
filipe oliveira authored
This Commit pushes forward the observability on overall error statistics and command statistics within redis-server: It extends INFO COMMANDSTATS to have - failed_calls in - so we can keep track of errors that happen from the command itself, broken by command. - rejected_calls - so we can keep track of errors that were triggered outside the commmand processing per se Adds a new section to INFO, named ERRORSTATS that enables keeping track of the different errors that occur within redis ( within processCommand and call ) based on the reply Error Prefix ( The first word after the "-", up to the first space ). This commit also fixes RM_ReplyWithError so that it can be correctly identified as an error reply.
-
- 25 Dec, 2020 1 commit
-
-
Itamar Haber authored
Adds: `L/RPOP <key> [count]` Implements no. 2 of the following strategies: 1. Loop on listTypePop - this would result in multiple calls for memory freeing and allocating (see https://github.com/redis/redis/pull/8179/commits/769167a079b0e110d28e4a8099dce1ecd45682b5) 2. Iterate the range to build the reply, then call quickListDelRange - this requires two iterations and **is the current choice** 3. Refactor quicklist to have a pop variant of quickListDelRange - probably optimal but more complex Also: * There's a historical check for NULL after calling listTypePop that was converted to an assert. * This refactors common logic shared between LRANGE and the new form of LPOP/RPOP into addListRangeReply (adds test for b/w compat) * Consequently, it may have made sense to have `LRANGE l -1 -2` and `LRANGE l 9 0` be legit and return a reverse reply. Due to historical reasons that would be, however, a breaking change. * Added minimal comments to existing commands to adhere to the style, make core dev life easier and get commit karma, naturally.
-
- 24 Dec, 2020 2 commits
-
-
Oran Agra authored
The commit deals with the syncWithMaster and the ugly state machine in it. It attempts to clean it a bit, but more importantly it uses pipeline for part of the work (rather than 7 round trips, we now have 4). i.e. the connect and PING are separate, then AUTH + 3 REPLCONF in one pipeline, and finally the PSYNC (must be separate since the master has to have an empty output buffer).
-
Madelyn Olson authored
Cleanup key tracking documentation, always cleanup the tracking table, and free the tracking table in an async manner when applicable.
-
- 22 Dec, 2020 2 commits
-
-
Oran Agra authored
Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216) In the distant history there was only the read flag for commands, and whatever command that didn't have the read flag was a write one. Then we added the write flag, but some portions of the code still used !read Also some commands that don't work on the keyspace at all, still have the read flag. Changes in this commit: 1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE 2. EXEC command used to decides if it should propagate a MULTI by looking at the command flags (!read & !admin). When i was about to change it to look at the write flag instead, i realized that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and SCRIPT, all 3 are not marked as either a read command or a write one (as they should), but all 3 are calling forceCommandPropagation. So instead of introducing a new flag to denote a command that "writes" but not into the keyspace, and still needs propagation, i decided to rely on the forceCommandPropagation, and just fix the code to propagate MULTI when needed rather than depending on the command flags at all. The implication of my change then is that now it won't decide to propagate MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and other commands which are neither read nor write. 3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in cluster.c to look at !write rather than read flag. This should have no implications, since these code paths are only reachable for commands which access keys, and these are always marked as either read or write. This commit improve MULTI propagation tests, for modules and a bunch of other special cases, all of which used to pass already before that commit. the only one that test change that uncovered a change of behavior is the one that DELs a non-existing key, it used to propagate an empty multi-exec block, and no longer does.
-
sundb authored
* Fix some redundancy use of semicolon in do-while macros
-