- 28 Feb, 2023 2 commits
-
-
uriyage authored
In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation size in order to: > reduce the need for realloc calls by making the sds implicitly take over the internal fragmentation This change was done most sds functions, excluding `sdsRemoveFreeSpace` and `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer) we call sdsRemoveFreeSpace when we see excessive free space and want to trim it. so if we don't trim it exactly to size, the caller may still see excessive free space and call it again and again. However, this resulted in some excessive calls to realloc, even when there's no need and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13). It turns out that a call for realloc with jemalloc can be expensive even if it ends up doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid the call for realloc. in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common code. the difference between them was that sdsResize would avoid using SDS_TYPE_5, since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace would permit using SDS_TYPE_5 and get an optimal memory consumption. now both methods take a `would_regrow` argument that makes it more explicit. the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both. The new test that was added to cover this concern used to pass before this PR as well, this PR is just a performance optimization and cleanup. Benchmark: `redis-benchmark -c 100 -t set -d 512 -P 10 -n 100000000` on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs). some 4.5% improvement. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit 46393f98) (cherry picked from commit b12eeccddd9318a5d97a5aee2dad88999dfad53f)
-
Oran Agra authored
Turns out that a fork child calling getExpire while persisting keys (and possibly also a result of some module fork tasks) could cause dictFind to do incremental rehashing in the child process, which is both a waste of time, and also causes COW harm. (cherry picked from commit 2bec254d) (cherry picked from commit 3e82bdf7)
-
- 12 Dec, 2022 2 commits
-
-
Oran Agra authored
During a diskless sync, if the master main process crashes, the child would have hung in `write`. This fix closes the read fd on the child side, so that if the parent crashes, the child will get a write error and exit. This change also fixes disk-based replication, BGSAVE and AOFRW. In that case the child wouldn't have been hang, it would have just kept running until done which may be pointless. There is a certain degree of risk here. in case there's a BGSAVE child that could maybe succeed and the parent dies for some reason, the old code would have let the child keep running and maybe succeed and avoid data loss. On the other hand, if the parent is restarted, it would have loaded an old rdb file (or none), and then the child could reach the end and rename the rdb file (data conflicting with what the parent has), or also have a race with another BGSAVE child that the new parent started. Note that i removed a comment saying a write error will be ignored in the child and handled by the parent (this comment was very old and i don't think relevant). (cherry picked from commit ccaef5c9)
-
- 27 Apr, 2022 5 commits
-
-
filipe oliveira authored
Optimization: Use either monotonic or wall-clock to measure command execution time, to regain up to 4% execution time (#10502) In #7491 (part of redis 6.2), we started using the monotonic timer instead of mstime to measure command execution time for stats, apparently this meant sampling the clock 3 times per command rather than two (wince we also need the wall-clock time). In some cases this causes a significant overhead. This PR fixes that by avoiding the use of monotonic timer, except for the cases were we know it should be extremely fast. This PR also adds a new INFO field called `monotonic_clock` that shows which clock redis is using. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit 3cd8baf6)
-
filipe oliveira authored
Avoid sprintf/ll2string on setDeferredAggregateLen()/addReplyLongLongWithPrefix() when we can used shared objects. In some pipelined workloads this achieves about 10% improvement. Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit b857928b)
-
Moti Cohen authored
As Sentinel relies upon consensus algorithm, all sentinel instances, randomize a time to initiate their next attempt to become the leader of the group. But time after time, all raffled the same value. The problem is in the line `srand(time(NULL)^getpid())` such that all spinned up containers get same time (in seconds) and same pid which is always 1. Added material `tv_usec` and verify that even consecutive calls brings different values and makes the difference. (cherry picked from commit 52b2fbe9)
-
Oran Agra authored
Following #9483 the daily CI exposed a few problems. * The cluster creation code (uses redis-cli) is complicated to test with TLS enabled. for now i'm just skipping them since the tests we run there don't really need that kind of coverage * cluster port binding failures note that `find_available_port` already looks for a free cluster port but the code in `wait_server_started` couldn't detect the failure of binding (the text it greps for wasn't found in the log) (cherry picked from commit 7d6744c7)
-
- 04 Oct, 2021 4 commits
-
-
yvette903 authored
A write request may be paused unexpectedly because `server.client_pause_end_time` is old. **Recreate this:** redis-cli -p 6379 127.0.0.1:6379> client pause 500000000 write OK 127.0.0.1:6379> client unpause OK 127.0.0.1:6379> client pause 10000 write OK 127.0.0.1:6379> set key value The write request `set key value` is paused util the timeout of 500000000 milliseconds was reached. **Fix:** reset `server.client_pause_end_time` = 0 in `unpauseClients` (cherry picked from commit f560531d)
-
zhaozhao.zz authored
When a replica paused, it would not apply any commands event the command comes from master, if we feed the non-applied command to replication stream, the replication offset would be wrong, and data would be lost after failover(since replica's `master_repl_offset` grows but command is not applied). To fix it, here are the changes: * Don't update replica's replication offset or propagate commands to sub-replicas when it's paused in `commandProcessed`. * Show `slave_read_repl_offset` in info reply. * Add an assert to make sure master client should never be blocked unless pause or module (some modules may use block way to do background (parallel) processing and forward original block module command to the replica, it's not a good way but it can work, so the assert excludes module now, but someday in future all modules should rewrite block command to propagate like what `BLPOP` does). (cherry picked from commit 1b83353d)
-
guybe7 authored
1. MIGRATE has a potnetial key arg in argv[3]. It should be reflected in the command table. 2. getKeysUsingCommandTable should never free getKeysResult, it is always freed by the caller) The reason we never encountered this double-free bug is that almost always getKeysResult uses the statis buffer and doesn't allocate a new one. (cherry picked from commit 6aa2285e)
-
Oran Agra authored
This change sets a low limit for multibulk and bulk length in the protocol for unauthenticated connections, so that they can't easily cause redis to allocate massive amounts of memory by sending just a few characters on the network. The new limits are 10 arguments of 16kb each (instead of 1m of 512mb)
-
- 21 Jul, 2021 1 commit
-
-
perryitay authored
There are two issues fixed in this commit: 1. we want to fail the EXEC command in case there is a watched key that's logically expired but not yet deleted by active expire or lazy expire. 2. we saw that currently cache time is update in every `call()` (including nested calls), this time is being also being use for the isKeyExpired comparison, we want to update the cache time only in the first call (execCommand) Co-authored-by:
Oran Agra <oran@redislabs.com> (cherry picked from commit ac8b1df8)
-
- 01 Jun, 2021 2 commits
-
-
Madelyn Olson authored
Redact commands that include sensitive data from slowlog and monitor (cherry picked from commit a59e75a4)
-
yoav-steinberg authored
When client breached the output buffer soft limit but then went idle, we didn't disconnect on soft limit timeout, now we do. Note this also resolves some sporadic test failures in due to Linux buffering data which caused tests to fail if during the test we went back under the soft COB limit. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
sundb <sundbcn@gmail.com> (cherry picked from commit 152fce5e)
-
- 03 May, 2021 1 commit
-
-
Oran Agra authored
This solves an issue reported in #8712 in which a replica would bypass the client write pause check and cause an assertion due to executing a write command during failover. The fact is that we don't expect replicas to execute any command other than maybe REPLCONF and PING, etc. but matching against the ADMIN command flag is insufficient, so instead i just block keyspace access for now. (cherry picked from commit 46f4ebbe)
-
- 19 Apr, 2021 1 commit
-
-
Wen Hui authored
When replica never successfully connect to master, server.repl_down_since will be initialized to 0, therefore, the info master_link_down_since_seconds was showing the current unix timestamp, which does not make much sense. This commit fixes the issue by showing master_link_down_since_seconds to -1. means the replica never connect to master before. This commit also resets this variable back to 0 when a replica is turned into a master, so that it'll behave the same if the master is later turned into a replica again. The implication of this change is that if some app is checking if the value is > 60 do something, like conclude the replica is stale, this could case harm (changing a big positive number with a small one).
-
- 13 Apr, 2021 1 commit
-
-
Wen Hui authored
-
- 11 Apr, 2021 1 commit
-
-
Wang Yuan authored
The bio aof fsync fd may be closed by main thread (AOFRW done handler) and even possibly reused for another socket, pipe, or file. This can can an EBADF or EINVAL fsync error, which will lead to -MISCONF errors failing all writes. We just ignore these errno because aof fsync did not really fail. We handle errno when fsyncing aof in bio, so we could know the real reason when users get -MISCONF Errors writing to the AOF file error Issue created with #8419
-
- 01 Apr, 2021 2 commits
-
-
Wang Yuan authored
In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail. In `cluster.c`, we now return error, the calling function already handles these write errors. In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed. In `rio.c`, we now treat fsync errors like we do for write errors. In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail. In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error, moreover also set INFO command field `aof_last_write_status` to error.
-
Wen Hui authored
The implications of this change is just that in the past when a config file was missing, in some cases it was exiting before printing the sever startup prints and sometimes after, and now it'll always exit before printing them.
-
- 30 Mar, 2021 1 commit
-
-
Jérôme Loyet authored
The 'sentinel replicas <master>' command will ignore replicas with `replica-announced` set to no. The goal of disabling the config setting replica-announced is to allow ghost replicas. The replica is in the cluster, synchronize with its master, can be promoted to master and is not exposed to sentinel clients. This way, it is acting as a live backup or living ghost. In addition, to prevent the replica to be promoted as master, set replica-priority to 0.
-
- 26 Mar, 2021 1 commit
-
-
Huang Zhw authored
Add publish channel permissions check in processCommand. processCommand didn't check publish channel permissions, so we can queue a publish command in a transaction. But when exec the transaction, it will fail with -NOPERM. We also union keys/commands/channels permissions check togegher in ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in publishCommand/subscribeCommand/psubscribeCommand. Always check permissions in processCommand/execCommand/ luaRedisGenericCommand.
-
- 25 Mar, 2021 1 commit
-
-
Oran Agra authored
* SLOWLOG didn't record anything for blocked commands because the client was reset and argv was already empty. there was a fix for this issue specifically for modules, now it works for all blocked clients. * The original command argv (before being re-written) was also reset before adding the slowlog on behalf of the blocked command. * Latency monitor is now updated regardless of the slowlog flags of the command or its execution (their purpose is to hide sensitive info from the slowlog, not hide the fact the latency happened). * Latency monitor now uses real_cmd rather than c->cmd (which may be different if the command got re-written, e.g. GEOADD) Changes: * Unify shared code between slowlog insertion in call() and updateStatsOnUnblock(), hopefully prevent future bugs from happening due to the later being overlooked. * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command processing. * Add a test for SLOWLOG and BLPOP Notes: - real_cmd == c->lastcmd, except inside MULTI and Lua. - blocked commands never happen in these cases (MULTI / Lua) - real_cmd == c->cmd, except for when the command is rewritten (e.g. GEOADD) - blocked commands (currently) are never rewritten - other than the command's CLIENT_PREVENT_LOGGING, and the execution flag CLIENT_PREVENT_LOGGING, other cases that we want to avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will be off when executed from execCommand that runs from an AOF)
-
- 24 Mar, 2021 2 commits
-
-
Qu Chen authored
Properly initialize variable to make valgrind happy in checkChildrenDone(). Removed usage for the obsolete wait3() and wait4() in favor of waitpid(), and properly check for the exit status code. (#8666)
-
yoav-steinberg authored
-
- 22 Mar, 2021 1 commit
-
-
Yossi Gottlieb authored
Reading CoW from /proc/<pid>/smaps can be slow with large processes on some platforms. This measures the time it takes to read CoW info and limits the duty cycle of future updates to roughly 1/100. As current_cow_size no longer represnets a current, fixed interval value there is also a new current_cow_size_age field that provides information about the age of the size value, in seconds.
-
- 16 Mar, 2021 3 commits
-
-
Huang Zhw authored
In certain scenario start_server may think it failed to start a redis server although it started successfully. in these cases, it'll not terminate it, and it'll remain running when the test is over. In start_server if config doesn't have bind (the minimal.conf in introspection.tcl), it will try to bind ipv4 and ipv6. One may success while other fails. It will output "Could not create server TCP listening socket". wait_server_started uses this message to check whether instance started successfully. So it will consider that it failed even though redis started successfully. Additionally, in some cases it wasn't clear to users why the server exited, since the warning message printed to the log, could in some cases be harmless, and in some cases fatal. This PR adds makes a clear distinction between a warning log message and a fatal one, and changes the test suite to look for the fatal message.
-
Yossi Gottlieb authored
listenToPort attempts to gracefully handle and ignore certain errors but does not store errno prior to logging, which in turn calls several libc functions that may overwrite errno. This has been discovered due to libmusl strftime() always returning with errno set to EINVAL, which resulted with docker-library/redis#273.
-
Madelyn Olson authored
Redact config set requirepass/masterauth/masteruser from slowlog in addition to showing ACL commands without sensitive values.
-
- 15 Mar, 2021 1 commit
-
-
guybe7 authored
1. moduleReplicateMultiIfNeeded should use server.in_eval like moduleHandlePropagationAfterCommandCallback 2. server.in_eval could have been set to 1 and not reset back to 0 (a lot of missed early-exits after in_eval is already 1) Note: The new assertions in processCommand cover (2) and I added two module tests to cover (1) Implications: If an EVAL that failed (and thus left server.in_eval=1) runs before a module command that replicates, the replication stream will contain MULTI (because moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL at this point) but not EXEC (because server.in_eval==1) This only affects modules as module.c the only user of server.in_eval. Affects versions 6.2.0, 6.2.1
-
- 14 Mar, 2021 1 commit
-
-
Guillem Jover authored
On a replica we do accept connections, even though commands accessing the database will operate in read-only mode. But the server is still already operational and processing commands. Not sending the readiness notification means that on a HA setup where the nodes all start as replicas (with replicaof in the config) with a replica that cannot connect to the master server and which might not come back in a predictable amount of time or at all, the service supervisor will end up timing out the service and terminating it, with no option to promote it to be the main instance. This seems counter to what the readiness notification is supposed to be signaling. Instead send the readiness notification when we start accepting commands, and then send the various server status changes as that. Fixes: commit 641c64ad Fixes: commit dfb598cf
-
- 11 Mar, 2021 1 commit
-
-
Oran Agra authored
same as DISCARD, this command (RESET) is about client state, not server state
-
- 10 Mar, 2021 3 commits
-
-
guybe7 authored
Bug 1: When a module ctx is freed moduleHandlePropagationAfterCommandCallback is called and handles propagation. We want to prevent it from propagating commands that were not replicated by the same context. Example: 1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3) 2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte 3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx. 4. moduleHandlePropagationAfterCommandCallback is called, calling alsoPropagates EXEC (Note: EXEC is still not written to socket), setting server.in_trnsaction = 0 5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now we have nested MULTI calls, which is no good) and then cmd3 We must prevent RM_Call(cmd2) from resetting server.in_transaction. REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose. Bug 2: Fix issues with nested RM_Call where some have '!' and some don't. Example: 1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!') 2. module2.bar internally calls RM_Call of INCR with '!' 3. at the end of module1.foo we call RM_ReplicateVerbatim We want the replica/AOF to see only module1.foo and not the INCR from module2.bar Introduced a global replication_allowed flag inside RM_Call to determine whether we need to replicate or not (even if '!' was specified) Other changes: Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec just for better readability
-
Oran Agra authored
`master_sync_perc` and `loading_loaded_perc` don't have that sign, and i think the info field should be a raw floating point number (the name suggests its units). we already have `used_memory_peak_perc` and `used_memory_dataset_perc` which do add the `%` sign, but: 1) i think it was a mistake but maybe too late to fix now, and maybe not too late to fix for `current_fork_perc` 2) it is more important to be consistent with the two other "progress" "prec" metrics, and not with the "utilization" metric.
-
sundb authored
1. Add `redis-server test all` support to run all tests. 2. Add redis test to daily ci. 3. Add `--accurate` option to run slow tests for more iterations (so that by default we run less cycles (shorter time, and less prints). 4. Move dict benchmark to REDIS_TEST. 5. fix some leaks in tests 6. make quicklist tests run on a specific fill set of options rather than huge ranges 7. move some prints in quicklist test outside their loops to reduce prints 8. removing sds.h from dict.c since it is now used in both redis-server and redis-cli (uses hiredis sds)
-
- 07 Mar, 2021 1 commit
-
-
Huang Zhw authored
-
- 03 Mar, 2021 1 commit
-
-
yoav-steinberg authored
When no connected clients variables stopped being updated instead of being zeroed over time. The other changes are cleanup and optimization (avoiding an unnecessary division per client)
-
- 01 Mar, 2021 1 commit
-
-
YaacovHazan authored
Add ability to modify port, tls-port and bind configurations by CONFIG SET command. To simplify the code and make it cleaner, a new structure added, socketFds, which contains the file descriptors array and its counter, and used for TCP, TLS and Cluster sockets file descriptors.
-