- 09 Sep, 2020 2 commits
-
-
Yossi Gottlieb authored
This is a catch-all test to confirm that that rewrite produces a valid output for all parameters and that this process does not introduce undesired configuration changes.
-
- 06 Sep, 2020 1 commit
-
-
Oran Agra authored
Starting redis 6.0 and the changes we made to the diskless master to be suitable for TLS, I made the master avoid reaping (wait3) the pid of the child until we know all replicas are done reading their rdb. I did that in order to avoid a state where the rdb_child_pid is -1 but we don't yet want to start another fork (still busy serving that data to replicas). It turns out that the solution used so far was problematic in case the fork child was being killed (e.g. by the kernel OOM killer), in that case there's a chance that we currently disabled the read event on the rdb pipe, since we're waiting for a replica to become writable again. and in that scenario the master would have never realized the child exited, and the replica will remain hung too. Note that there's no mechanism to detect a hung replica while it's in rdb transfer state. The solution here is to add another pipe which is used by the parent to tell the child it is safe to exit. this mean that when the child exits, for whatever reason, it is safe to reap it. Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was part of #6271 (Accelerate diskless master connections) but was dropped when that PR was rebased after the TLS fork/pipe changes (5a477946). Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK has chance to detect that the child exited, it should be the one to call it so that we don't have to wait for cron (server.hz) to do that.
-
- 03 Sep, 2020 1 commit
-
-
Oran Agra authored
During long running scripts or loading RDB/AOF, we may need to do some defragging. Since processEventsWhileBlocked is called periodically at unknown intervals, and many cron jobs either depend on run_with_period (including active defrag), or rely on being called at server.hz rate (i.e. active defrag knows ho much time to run by looking at server.hz), the whileBlockedCron may have to run a loop triggering the cron jobs in it (currently only active defrag) several times. Other changes: - Adding a test for defrag during aof loading. - Changing key-load-delay config to take negative values for fractions of a microsecond sleep
-
- 02 Sep, 2020 1 commit
-
-
Oran Agra authored
When redis isn't configured to have a log file, having these prints before damonization puts them in the calling process stdout rather than /dev/null
-
- 28 Aug, 2020 1 commit
-
-
Jim Brunner authored
Update adds a general source for retrieving a monotonic time. In addition, AE has been updated to utilize the new monotonic clock for timer processing. This performance improvement is **not** enabled in a default build due to various H/W compatibility concerns, see README.md for details. It does however change the default use of gettimeofday with clock_gettime and somewhat improves performance. This update provides the following 1. An interface for retrieving a monotonic clock. getMonotonicUs returns a uint64_t (aka monotime) with the number of micro-seconds from an arbitrary point. No more messing with tv_sec/tv_usec. Simple routines are provided for measuring elapsed milli-seconds or elapsed micro-seconds (the most common use case for a monotonic timer). No worries about time moving backwards. 2. High-speed assembler implementation for x86 and ARM. The standard method for retrieving the monotonic clock is POSIX.1b (1993): clock_gettime(CLOCK_MONOTONIC, timespec*). However, most modern processors provide a constant speed instruction clock which can be retrieved in a fraction of the time that it takes to call clock_gettime. For x86, this is provided by the RDTSC instruction. For ARM, this is provided by the CNTVCT_EL0 instruction. As a compile-time option, these high-speed timers can be chosen. (Default is POSIX clock_gettime.) 3. Refactor of event loop timers. The timer processing in ae.c has been refactored to use the new monotonic clock interface. This results in simpler/cleaner logic and improved performance.
-
- 27 Aug, 2020 3 commits
-
-
Oran Agra authored
65a3307b (released in 6.0.6) has a side effect, when processCommand rejects a command with pre-made shared object error string, it trims the newlines from the end of the string. if that string is later used with addReply, the newline will be missing, breaking the protocol, and leaving the client hung. It seems that the only scenario which this happens is when replying with -LOADING to some command, and later using that reply from the CONFIG SET command (still during loading). this will result in hung client. Refactoring the code in order to avoid trimming these newlines from shared string objects, and do the newline trimming only in other cases where it's needed. Co-authored-by:
Guy Benoish <guy.benoish@redislabs.com>
-
Oran Agra authored
During a long AOF or RDB loading, the memory stats were not updated, and INFO would return stale data, specifically about fragmentation and RSS. In the past some of these were sampled directly inside the INFO command, but were moved to cron as an optimization. This commit introduces a concept of loadingCron which should take some of the responsibilities of serverCron. It attempts to limit it's rate to approximately the server Hz, but may not be very accurate. In order to avoid too many system call, we use the cached ustime, and also make sure to update it in both AOF loading and RDB loading inside processEventsWhileBlocked (it seems AOF loading was missing it).
-
valentinogeron authored
If the server gets MULTI command followed by only read commands, and right before it gets the EXEC it reaches OOM, the client will get OOM response. So, from now on, it will get OOM response only if there was at least one command that was tagged with `use-memory` flag
-
- 24 Aug, 2020 1 commit
-
-
Wang Yuan authored
unlike printf, sdscatfmt doesn't take %d
-
- 21 Aug, 2020 1 commit
-
-
Wen Hui authored
-
- 20 Aug, 2020 1 commit
-
-
杨博东 authored
After fork, the child process(redis-aof-rewrite) will get the fd opened by the parent process(redis), when redis killed by kill -9, it will not graceful exit(call prepareForShutdown()), so redis-aof-rewrite thread may still alive, the fd(lock) will still be held by redis-aof-rewrite thread, and redis restart will fail to get lock, means fail to start. This issue was causing failures in the cluster tests in github actions. Co-authored-by:Oran Agra <oran@redislabs.com>
-
- 18 Aug, 2020 1 commit
-
-
Oran Agra authored
Since users often post just the crash log in github issues, the log print that's above it is missing. No reason not to include the size in the panic message itself.
-
- 12 Aug, 2020 3 commits
-
-
RemRain authored
-
Yossi Gottlieb authored
Add Linux kernel OOM killer control option. This adds the ability to control the Linux OOM killer oom_score_adj parameter for all Redis processes, depending on the process role (i.e. master, replica, background child). A oom-score-adj global boolean flag control this feature. In addition, specific values can be configured using oom-score-adj-values if additional tuning is required.
-
HarveyLiu authored
and a misspell in rax.c
-
- 11 Aug, 2020 1 commit
-
-
Tyson Andre authored
This is a rebased version of #3078 originally by shaharmor with the following patches by TysonAndre made after rebasing to work with the updated C API: 1. Add 2 more unit tests (wrong argument count error message, integer over 64 bits) 2. Use addReplyArrayLen instead of addReplyMultiBulkLen. 3. Undo changes to src/help.h - for the ZMSCORE PR, I heard those should instead be automatically generated from the redis-doc repo if it gets updated Motivations: - Example use case: Client code to efficiently check if each element of a set of 1000 items is a member of a set of 10 million items. (Similar to reasons for working on #7593) - HMGET and ZMSCORE already exist. This may lead to developers deciding to implement functionality that's best suited to a regular set with a data type of sorted set or hash map instead, for the multi-get support. Currently, multi commands or lua scripting to call sismember multiple times would almost definitely be less efficient than a native smismember for the following reasons: - Need to fetch the set from the string every time instead of reusing the C pointer. - Using pipelining or multi-commands would result in more bytes sent and received by the client for the repeated SISMEMBER KEY sections. - Need to specially encode the data and decode it from the client for lua-based solutions. - Proposed solutions using Lua or SADD/SDIFF could trigger writes to memory, which is undesirable on a redis replica server or when commands get replicated to replicas. Co-Authored-By:
Shahar Mor <shahar@peer5.com> Co-Authored-By:
Tyson Andre <tysonandre775@hotmail.com>
-
- 08 Aug, 2020 2 commits
-
-
WuYunlong authored
-
Wang Yuan authored
Before this fix we where attempting to select a db before creating db the DB, see: #7323 This issue doesn't seem to have any implications, since the selected DB index is 0, the db pointer remains NULL, and will later be correctly set before using this dummy client for the first time. As we know, we call 'moduleInitModulesSystem()' before 'initServer()'. We will allocate memory for server.db in 'initServer', but we call 'createClient()' that will call 'selectDb()' in 'moduleInitModulesSystem()', before the databases where created. Instead, we should call 'createClient()' for moduleFreeContextReusedClient after 'initServer()'.
-
- 06 Aug, 2020 2 commits
-
-
Oran Agra authored
Diskless master has some inherent latencies. 1) fork starts with delay from cron rather than immediately 2) replica is put online only after an ACK. but the ACK was sent only once a second. 3) but even if it would arrive immediately, it will not register in case cron didn't yet detect that the fork is done. Besides that, when a replica disconnects, it doesn't immediately attempts to re-connect, it waits for replication cron (one per second). in case it was already online, it may be important to try to re-connect as soon as possible, so that the backlog at the master doesn't vanish. In case it disconnected during rdb transfer, one can argue that it's not very important to re-connect immediately, but this is needed for the "diskless loading short read" test to be able to run 100 iterations in 5 seconds, rather than 3 (waiting for replication cron re-connection) changes in this commit: 1) sync command starts a fork immediately if no sync_delay is configured 2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron) 3) when a replica unexpectedly disconnets, it immediately tries to re-connect rather than waiting 1s 4) when when a child exits, if there is another replica waiting, we spawn a new one right away, instead of waiting for 1s replicationCron. 5) added a call to connectWithMaster from replicationSetMaster. which is called from the REPLICAOF command but also in 3 places in cluster.c, in all of these the connection attempt will now be immediate instead of delayed by 1 second. side note: we can add a call to rdbPipeReadHandler in replconfCommand when getting a REPLCONF ACK from the replica to solve a race where the replica got the entire rdb and EOF marker before we detected that the pipe was closed. in the test i did see this race happens in one about of some 300 runs, but i concluded that this race is unlikely in real life (where the replica is on another host and we're more likely to first detect the pipe was closed. the test runs 100 iterations in 3 seconds, so in some cases it'll take 4 seconds instead (waiting for another REPLCONF ACK). Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave Now that CheckChildrenDone is calling the new replicationStartPendingFork (extracted from serverCron) there's actually no need to call startBgsaveForReplication from updateSlavesWaitingForBgsave anymore, since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is calling replicationStartPendingFork that handles that anyway. The code in updateSlavesWaitingForBgsave had a bug in which it ignored repl-diskless-sync-delay, but removing that code shows that this bug was hiding another bug, which is that the max_idle should have used >= and not >, this one second delay has a big impact on my new test.
-
Oran Agra authored
This makes it possible to add tests that generate assertions, and run them with valgrind, making sure that there are no memory violations prior to the assertion. New config options: - crash-log-enabled - can be disabled for cleaner core dumps - crash-memcheck-enabled - useful for faster termination after a crash - use-exit-on-panic - to be used by the test suite so that valgrind can detect leaks and memory corruptions Other changes: - Crash log is printed even on system that dont HAVE_BACKTRACE, i.e. in both SIGSEGV and assert / panic - Assertion and panic won't print registers and code around EIP (which was useless), but will do fast memory test (which may still indicate that the assertion was due to memory corrpution) I had to reshuffle code in order to re-use it, so i extracted come code into function without actually doing any changes to the code: - logServerInfo - logModulesInfo - doFastMemoryTest (with the exception of it being conditional) - dumpCodeAroundEIP changes to the crash report on segfault: - logRegisters is called right after the stack trace (before info) done just in order to have more re-usable code - stack trace skips the first two items on the stack (the crash log and signal handler functions)
-
- 04 Aug, 2020 1 commit
-
-
Tyson Andre authored
Syntax: `ZMSCORE KEY MEMBER [MEMBER ...]` This is an extension of #2359 amended by Tyson Andre to work with the changed unstable API, add more tests, and consistently return an array. - It seemed as if it would be more likely to get reviewed after updating the implementation. Currently, multi commands or lua scripting to call zscore multiple times would almost definitely be less efficient than a native ZMSCORE for the following reasons: - Need to fetch the set from the string every time instead of reusing the C pointer. - Using pipelining or multi-commands would result in more bytes sent by the client for the repeated `ZMSCORE KEY` sections. - Need to specially encode the data and decode it from the client for lua-based solutions. - The fastest solution I've seen for large sets(thousands or millions) involves lua and a variadic ZADD, then a ZINTERSECT, then a ZRANGE 0 -1, then UNLINK of a temporary set (or lua). This is still inefficient. Co-authored-by:Tyson Andre <tysonandre775@hotmail.com>
-
- 29 Jul, 2020 1 commit
-
-
Arun Ranganathan authored
Co-authored-by:Oran Agra <oran@redislabs.com>
-
- 27 Jul, 2020 1 commit
-
-
Yossi Gottlieb authored
Initialize and configure OpenSSL even when tls-port is not used, because we may still have tls-cluster or tls-replication. Also, make sure to reconfigure OpenSSL when these parameters are changed as TLS could have been enabled for the first time.
-
- 22 Jul, 2020 1 commit
-
-
Madelyn Olson authored
-
- 21 Jul, 2020 1 commit
-
-
WuYunlong authored
Before that PR, processCommand() did not notice that cmd could be a module command in which case getkeys_proc member has a different meaning. The outcome was that a module command which doesn't take any key names in its arguments (similar to SLOWLOG) would be handled as if it might have key name arguments (similar to MEMORY), would consider cluster redirect but will end up with 0 keys after an excessive call to getKeysFromCommand, and eventually do the right thing.
-
- 20 Jul, 2020 1 commit
-
-
WuYunlong authored
Before this commit, processCommand() did not notice that cmd could be a module command which declared `getkeys-api` and handled it for the purpose of cluster redirect it as if it doesn't use any keys. This commit fixed it by reusing the codes in addReplyCommand().
-
- 15 Jul, 2020 1 commit
-
-
dmurnane authored
Co-authored-by:Daniel Murnane <dmurnane@eitccorp.com>
-
- 23 Jun, 2020 1 commit
-
-
Oran Agra authored
In order to support the use of multi-exec in pipeline, it is important that MULTI and EXEC are never rejected and it is easy for the client to know if the connection is still in multi state. It was easy to make sure MULTI and DISCARD never fail (done by previous commits) since these only change the client state and don't do any actual change in the server, but EXEC is a different story. Since in the past, it was possible for clients to handle some EXEC errors and retry the EXEC, we now can't affort to return any error on EXEC other than EXECABORT, which now carries with it the real reason for the abort too. Other fixes in this commit: - Some checks that where performed at the time of queuing need to be re- validated when EXEC runs, for instance if the transaction contains writes commands, it needs to be aborted. there was one check that was already done in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF, -NOREPLICAS, -MASTERDOWN - When a command is rejected by processCommand it was rejected with addReply, which was not recognized as an error in case the bad command came from the master. this will enable to count or MONITOR these errors in the future. - make it easier for tests to create additional (non deferred) clients. - add tests for the fixes of this commit.
-
- 17 Jun, 2020 1 commit
-
-
Tomasz Poradowski authored
- enforcing of SHUTDOWN_NOSAVE flag in one place to make it consitent when running in Sentinel mode
-
- 10 Jun, 2020 2 commits
-
-
antirez authored
-
Paul Spooren authored
The `LRANK` command returns the index (position) of a given element within a list. Using the `direction` argument it is possible to specify going from head to tail (acending, 1) or from tail to head (decending, -1). Only the first found index is returend. The complexity is O(N). When using lists as a queue it can be of interest at what position a given element is, for instance to monitor a job processing through a work queue. This came up within the Python `rq` project which is based on Redis[0]. [0]: https://github.com/rq/rq/issues/1197 Signed-off-by:
Paul Spooren <mail@aparcar.org>
-
- 08 Jun, 2020 1 commit
-
-
Oran Agra authored
Much like MULTI/EXEC/DISCARD, the WATCH and UNWATCH are not actually operating on the database or server state, but instead operate on the client state. the client may send them all in one long pipeline and check all the responses only at the end, so failing them may lead to a mismatch between the client state on the server and the one on the client end, and execute the wrong commands (ones that were meant to be discarded) the watched keys are not actually stored in the client struct, but they are in fact part of the client state. for instance, they're not cleared or moved in SWAPDB or FLUSHDB.
-
- 27 May, 2020 1 commit
-
-
antirez authored
After a closer look, the Redis core devleopers all believe that this was too fragile, caused many bugs that we didn't expect and that were very hard to track. Better to find an alternative solution that is simpler.
-
- 14 May, 2020 2 commits
- 12 May, 2020 1 commit
-
-
antirez authored
-
- 11 May, 2020 1 commit
-
-
Oran Agra authored
This bug was introduced by a recent change in which readQueryFromClient is using freeClientAsync, and despite the fact that now freeClientsInAsyncFreeQueue is in beforeSleep, that's not enough since it's not called during loading in processEventsWhileBlocked. furthermore, afterSleep was called in that case but beforeSleep wasn't. This bug also caused slowness sine the level-triggered mode of epoll kept signaling these connections as readable causing us to keep doing connRead again and again for ll of these, which keep accumulating. now both before and after sleep are called, but not all of their actions are performed during loading, some are only reserved for the main loop. fixes issue #7215
-
- 05 May, 2020 2 commits