- 19 May, 2021 1 commit
-
-
Madelyn Olson authored
Redact commands that include sensitive data from slowlog and monitor
-
- 25 Apr, 2021 1 commit
-
-
Huang Zhw authored
-
- 19 Apr, 2021 1 commit
-
-
Wen Hui authored
When replica never successfully connect to master, server.repl_down_since will be initialized to 0, therefore, the info master_link_down_since_seconds was showing the current unix timestamp, which does not make much sense. This commit fixes the issue by showing master_link_down_since_seconds to -1. means the replica never connect to master before. This commit also resets this variable back to 0 when a replica is turned into a master, so that it'll behave the same if the master is later turned into a replica again. The implication of this change is that if some app is checking if the value is > 60 do something, like conclude the replica is stale, this could case harm (changing a big positive number with a small one).
-
- 16 Apr, 2021 1 commit
-
-
guybe7 authored
Fixes #8797
-
- 15 Apr, 2021 1 commit
-
-
guybe7 authored
Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork child so that the parent is the one sending data to the replicas. This mechanism has an issue in which a hung replica will cause the master to wait for it to read the data sent to it forever, thus preventing the fork child from terminating and preventing the creations of any other forks. This PR adds a timeout mechanism, much like the ACK-based timeout, we disconnect replicas that aren't reading the RDB file fast enough.
-
- 25 Mar, 2021 1 commit
-
-
hzzb authored
REPLICAOF actually specifies which master to attach.
-
- 14 Mar, 2021 2 commits
-
-
Huang Zhw authored
-
Guillem Jover authored
On a replica we do accept connections, even though commands accessing the database will operate in read-only mode. But the server is still already operational and processing commands. Not sending the readiness notification means that on a HA setup where the nodes all start as replicas (with replicaof in the config) with a replica that cannot connect to the master server and which might not come back in a predictable amount of time or at all, the service supervisor will end up timing out the service and terminating it, with no option to promote it to be the main instance. This seems counter to what the readiness notification is supposed to be signaling. Instead send the readiness notification when we start accepting commands, and then send the various server status changes as that. Fixes: commit 641c64ad Fixes: commit dfb598cf
-
- 21 Feb, 2021 1 commit
-
-
Yossi Gottlieb authored
Originally this was limited to IPv6 address length, but effectively it has been used for host names and now that Sentinel accepts that as well we need to be able to store full hostnames. Fixes #8507
-
- 09 Feb, 2021 1 commit
-
-
Madelyn Olson authored
Moved most static strings into the shared structure
-
- 28 Jan, 2021 1 commit
-
-
Allen Farris authored
Implement FAILOVER command, which coordinates failover between the server and one of its replicas.
-
- 27 Jan, 2021 1 commit
-
-
Wang Yuan authored
In some scenarios, such as remote backup, we only want to get remote redis server db snapshot. Currently, redis-cli acts as a replica and sends SYNC to redis, but redis still accumulates replication buffer in the replica client output buffer, that may result in using vast memory, or failing to transfer RDB because of client-output-buffer-limit. In this commit, we add 'replconf rdb-only 0|1', redis doesn't send incremental replication buffer to them if they send 'replconf rdb-only 1', so we can reduce used memory and improve success of getting RDB.
-
- 08 Jan, 2021 1 commit
-
-
Madelyn Olson authored
Implementation of client pause WRITE and client unpause
-
- 07 Jan, 2021 1 commit
-
-
YaacovHazan authored
This is a refactory commit, isn't suppose to have any actual impact. it does the following: - keep just one server struct fork child pid variable instead of 3 - have one server struct variable indicating the purpose of the current fork child. - redisFork is now responsible of updating the server struct with the pid, which means it can be the one that calls updateDictResizePolicy - move child info pipe handling into redisFork instead of having them repeated outside - there are two classes of fork purposes, mutually exclusive group (AOF, RDB, Module), and one that can create several forks to coexist in parallel (LDB, but maybe Modules some day too, Module API allows for that). - minor fix to killRDBChild: unlike killAppendOnlyChild and TerminateModuleForkChild, the killRDBChild doesn't clear the pid variable or call wait4, so checkChildrenDone does the cleanup for it. This commit removes the explicit calls to rdbRemoveTempFile, closeChildInfoPipe, updateDictResizePolicy, which didn't do any harm, but where unnecessary.
-
- 03 Jan, 2021 1 commit
-
-
Oran Agra authored
In #7726 (part of 6.2), we added a mechanism for whileBlockedCron, this mechanism has an assertion to make sure the timestamp in whileBlockedCron was always set correctly before the blocking operation starts. I now found (thanks to our CI) two bugs in that area: 1) CONFIG RESETSTAT (if it was allowed during loading) would have cleared this var 2) the call stopLoading (which calls whileBlockedCron) was made too early, while the rio is still in use, in which case the update_cksum (rdbLoadProgressCallback) may still be called and whileBlockedCron can assert.
-
- 24 Dec, 2020 4 commits
-
-
Oran Agra authored
The commit deals with the syncWithMaster and the ugly state machine in it. It attempts to clean it a bit, but more importantly it uses pipeline for part of the work (rather than 7 round trips, we now have 4). i.e. the connect and PING are separate, then AUTH + 3 REPLCONF in one pipeline, and finally the PSYNC (must be separate since the master has to have an empty output buffer).
-
Oran Agra authored
This is just a refactoring commit. This function was never actually used as a synchronous (do both send or receive), it was always used only ine one of the two modes, which meant it has to take extra arguments that are not relevant for the other. Besides that, a tool that sends a synchronous command, it not something we want in our toolbox (synchronous IO in single threaded app is evil). sendSynchronousCommand was now refactored into separate sending and receiving APIs, and the sending part has two variants, one taking vaargs, and the other taking argc+argv (and an optional length array which means you can use binary sds strings).
-
Madelyn Olson authored
Cleanup key tracking documentation, always cleanup the tracking table, and free the tracking table in an async manner when applicable.
-
Madelyn Olson authored
Properly throw errors for invalid replication stream and support https://github.com/redis/redis/pull/8217
-
- 17 Dec, 2020 1 commit
-
-
Qu Chen authored
* Handle binary safe string for REQUIREPASS and MASTERAUTH directives.
-
- 02 Dec, 2020 1 commit
-
-
Wang Yuan authored
Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108) When replica diskless-load type is swapdb in cluster mode, we didn't backup keys to slots map, so we will lose keys to slots map if fail to sync. Now we backup keys to slots map at first, and restore it properly when fail. This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit). Co-authored-by:Oran Agra <oran@redislabs.com>
-
- 22 Nov, 2020 1 commit
-
-
Wang Yuan authored
If we enable diskless replication, set repl-diskless-sync-delay to 0, and master has non-rdb child process such as rewrite aof child, master will try to start to a new BGSAVE but fails immediately (before fork) when replicas ask for full synchronization, and master always fails to start a new BGSAVE and disconnects with replicas until non-rdb child process exists. this bug was introduced in #6271 (not yet released in 6.0.x)
-
- 17 Nov, 2020 1 commit
-
-
Meir Shpilraien (Spielrein) authored
Blocking command should not be used with MULTI, LUA, and RM_Call. This is because, the caller, who executes the command in this context, expects a reply. Today, LUA and MULTI have a special (and different) treatment to blocking commands: LUA - Most commands are marked with no-script flag which are checked when executing and command from LUA, commands that are not marked (like XREAD) verify that their blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag). MULTI - Command that is going to block, first verify that the client is not inside multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they return a result which is a match to the empty key with no timeout (for example blpop inside MULTI will act as lpop) For modules that perform RM_Call with blocking command, the returned results type is REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened. Disadvantages of the current state are: No unified approach, LUA, MULTI, and RM_Call, each has a different treatment Module can not safely execute blocking command (and get reply or error). Though It is true that modules are not like LUA or MULTI and should be smarter not to execute blocking commands on RM_Call, sometimes you want to execute a command base on client input (for example if you create a module that provides a new scripting language like javascript or python). While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to check if the command came from another module using RM_Call. So there is no way for a module to know not to block another module RM_Call execution. This commit adds a way to unify the treatment for blocking clients by introducing a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag turned on to signify that the client should not be blocked. A blocking command verifies that the flag is turned off before blocking. If a blocking command sees that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results which are matches to empty key with no timeout (as MULTI does today). The new flag is checked on the following commands: List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE, Zset blocking commands: BZPOPMIN, BZPOPMAX Stream blocking commands: XREAD, XREADGROUP SUBSCRIBE, PSUBSCRIBE, MONITOR In addition, the new flag is turned on inside the AOF client, we do not want to block the AOF client to prevent deadlocks and commands ordering issues (and there is also an existing assert in the code that verifies it). To keep backward compatibility on LUA, all the no-script flags on existing commands were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept. To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE). We added a special treatment on those commands to allow executing them on MULTI. The only backward compatibility issue that this PR introduces is that now MONITOR is not allowed inside MULTI. Tests were added to verify blocking commands are not blocking the client on LUA, MULTI, or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
Itamar Haber <itamar@redislabs.com>
-
- 28 Oct, 2020 1 commit
-
-
yoav-steinberg authored
Useful when you want to know through which bind address the client connected to the server in case of multiple bind addresses. - Adding `laddr` field to CLIENT list showing the local (bind) address. - Adding `LADDR` option to CLIENT KILL to kill all the clients connected to a specific local address. - Refactoring to share code.
-
- 13 Oct, 2020 1 commit
-
-
Wang Yuan authored
If we fail or stop to rewrite aof, we need to remove temporary aof. We also remove temporary rdb when replicas abort to receive rdb. But currently we delete them in main thread, to avoid blocking, we should use bg_unlink to remove them in a background thread. Btw, we have already used this way to removed child process temporary rdb.
-
- 25 Sep, 2020 1 commit
-
-
WuYunlong authored
We should sync temp DB file before renaming as rdb_fsync_range does not use flag `SYNC_FILE_RANGE_WAIT_AFTER`. Refer to `Linux Programmer's Manual`: SYNC_FILE_RANGE_WAIT_AFTER Wait upon write-out of all pages in the range after performing any write.
-
- 22 Sep, 2020 1 commit
-
-
Yossi Gottlieb authored
This happens only on diskless replicas when attempting to reconnect after failing to load an RDB file. It is more likely to occur with larger datasets. After reconnection is initiated, replicationEmptyDbCallback() may get called and try to write to an unconnected socket. This triggered another issue where the connection is put into an error state and the connect handler never gets called. The problem is a regression introduced by commit c17e597d.
-
- 17 Sep, 2020 1 commit
-
-
Wang Yuan authored
Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11 _Atomic to establish inter-thread synchronization without mutex. But the compiler that must supports C11 _Atomic can compile redis code, that brings a lot of inconvenience since some common platforms can't support by default such as CentOS7, so we want to implement redis atomic type to make it more portable. We have implemented our atomic variable for redis that only has 'relaxed' operations in src/atomicvar.h, so we implement some operations with 'sequentially-consistent', just like the default behavior of C11 _Atomic that can establish inter-thread synchronization. And we replace all uses of C11 _Atomic with redis atomic variable. Our implementation of redis atomic variable uses C11 _Atomic, __atomic or __sync macros if available, it supports most common platforms, and we will detect automatically which feature we use. In Makefile we use a dummy file to detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis code theoretically if your gcc version is not less than 4.1.2(starts to support __sync_xxx operations). Otherwise, we remove use mutex fallback to implement redis atomic variable for performance and test. You will get compiling errors if your compiler doesn't support all features of above. For cover redis atomic variable tests, we add other CI jobs that build redis on CentOS6 and CentOS7 and workflow daily jobs that run the tests on them. For them, we just install gcc by default in order to cover different compiler versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7. We restore the feature that we can test redis with Helgrind to find data race errors. But you need install Valgrind in the default path configuration firstly before running your tests, since we use macros in helgrind.h to tell Helgrind inter-thread happens-before relationship explicitly for avoiding false positives. Please open an issue on github if you find data race errors relate to this commit. Unrelated: - Fix redefinition of typedef 'RedisModuleUserChangedFunc' For some old version compilers, they will report errors or warnings, if we re-define function type.
-
- 10 Sep, 2020 1 commit
-
-
Oran Agra authored
List of squashed commits or PRs =============================== commit 66801ea Author: hwware <wen.hui.ware@gmail.com> Date: Mon Jan 13 00:54:31 2020 -0500 typo fix in acl.c commit 46f55db Author: Itamar Haber <itamar@redislabs.com> Date: Sun Sep 6 18:24:11 2020 +0300 Updates a couple of comments Specifically: * RM_AutoMemory completed instead of pointing to docs * Updated link to custom type doc commit 61a2aa0 Author: xindoo <xindoo@qq.com> Date: Tue Sep 1 19:24:59 2020 +0800 Correct errors in code comments commit a5871d1 Author: yz1509 <pro-756@qq.com> Date: Tue Sep 1 18:36:06 2020 +0800 fix typos in module.c commit 41eede7 Author: bookug <bookug@qq.com> Date: Sat Aug 15 01:11:33 2020 +0800 docs: fix typos in comments commit c303c84 Author: lazy-snail <ws.niu@outlook.com> Date: Fri Aug 7 11:15:44 2020 +0800 fix spelling in redis.conf commit 1e...
-
- 06 Sep, 2020 1 commit
-
-
Oran Agra authored
Starting redis 6.0 and the changes we made to the diskless master to be suitable for TLS, I made the master avoid reaping (wait3) the pid of the child until we know all replicas are done reading their rdb. I did that in order to avoid a state where the rdb_child_pid is -1 but we don't yet want to start another fork (still busy serving that data to replicas). It turns out that the solution used so far was problematic in case the fork child was being killed (e.g. by the kernel OOM killer), in that case there's a chance that we currently disabled the read event on the rdb pipe, since we're waiting for a replica to become writable again. and in that scenario the master would have never realized the child exited, and the replica will remain hung too. Note that there's no mechanism to detect a hung replica while it's in rdb transfer state. The solution here is to add another pipe which is used by the parent to tell the child it is safe to exit. this mean that when the child exits, for whatever reason, it is safe to reap it. Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was part of #6271 (Accelerate diskless master connections) but was dropped when that PR was rebased after the TLS fork/pipe changes (5a477946). Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK has chance to detect that the child exited, it should be the one to call it so that we don't have to wait for cron (server.hz) to do that.
-
- 19 Aug, 2020 1 commit
-
-
Wang Yuan authored
-
- 13 Aug, 2020 1 commit
-
-
caozb authored
Client that issued WAIT last will most likely have the highest replication offset, so imagine a probably common case where all clients are waiting for the same number of replicas. we prefer the loop to start from the last client (one waiting for the highest offset), so that the optimization in the function will call replicationCountAcksByOffset for each client until it found a good one, and stop calling it for the rest of the clients. the way the loop was implemented would mean that in such case it is likely to call replicationCountAcksByOffset for all clients. Note: the change from > to >= is not directly related to the above. Co-authored-by:曹正斌 <caozb@jiedaibao.com>
-
- 12 Aug, 2020 1 commit
-
-
Yossi Gottlieb authored
Add Linux kernel OOM killer control option. This adds the ability to control the Linux OOM killer oom_score_adj parameter for all Redis processes, depending on the process role (i.e. master, replica, background child). A oom-score-adj global boolean flag control this feature. In addition, specific values can be configured using oom-score-adj-values if additional tuning is required.
-
- 06 Aug, 2020 1 commit
-
-
Oran Agra authored
Diskless master has some inherent latencies. 1) fork starts with delay from cron rather than immediately 2) replica is put online only after an ACK. but the ACK was sent only once a second. 3) but even if it would arrive immediately, it will not register in case cron didn't yet detect that the fork is done. Besides that, when a replica disconnects, it doesn't immediately attempts to re-connect, it waits for replication cron (one per second). in case it was already online, it may be important to try to re-connect as soon as possible, so that the backlog at the master doesn't vanish. In case it disconnected during rdb transfer, one can argue that it's not very important to re-connect immediately, but this is needed for the "diskless loading short read" test to be able to run 100 iterations in 5 seconds, rather than 3 (waiting for replication cron re-connection) changes in this commit: 1) sync command starts a fork immediately if no sync_delay is configured 2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron) 3) when a replica unexpectedly disconnets, it immediately tries to re-connect rather than waiting 1s 4) when when a child exits, if there is another replica waiting, we spawn a new one right away, instead of waiting for 1s replicationCron. 5) added a call to connectWithMaster from replicationSetMaster. which is called from the REPLICAOF command but also in 3 places in cluster.c, in all of these the connection attempt will now be immediate instead of delayed by 1 second. side note: we can add a call to rdbPipeReadHandler in replconfCommand when getting a REPLCONF ACK from the replica to solve a race where the replica got the entire rdb and EOF marker before we detected that the pipe was closed. in the test i did see this race happens in one about of some 300 runs, but i concluded that this race is unlikely in real life (where the replica is on another host and we're more likely to first detect the pipe was closed. the test runs 100 iterations in 3 seconds, so in some cases it'll take 4 seconds instead (waiting for another REPLCONF ACK). Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave Now that CheckChildrenDone is calling the new replicationStartPendingFork (extracted from serverCron) there's actually no need to call startBgsaveForReplication from updateSlavesWaitingForBgsave anymore, since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is calling replicationStartPendingFork that handles that anyway. The code in updateSlavesWaitingForBgsave had a bug in which it ignored repl-diskless-sync-delay, but removing that code shows that this bug was hiding another bug, which is that the max_idle should have used >= and not >, this one second delay has a big impact on my new test.
-
- 31 Jul, 2020 1 commit
-
-
Oran Agra authored
besides, hooks test was time sensitive. when the replica managed to reconnect quickly after the client kill, the test would fail
-
- 20 Jul, 2020 1 commit
-
-
zhaozhao.zz authored
-
- 14 Jul, 2020 1 commit
-
-
Oran Agra authored
in case the rdb child failed, crashed or terminated unexpectedly redis would have marked the replica clients with repl_put_online_on_ack and then kill them only after a minute when no ack was received. it would not stream anything to these connections, so the only effect of this bug is a delay of 1 minute in the replicas attempt to re-connect.
-
- 06 Jun, 2020 2 commits
-
- 28 May, 2020 1 commit
-
-
antirez authored
-