- 14 Feb, 2023 1 commit
-
-
guybe7 authored
Starting from Redis 7.0 (#9890) we started wrapping everything a command propagates with MULTI/EXEC. The problem is that both SCAN and RANDOMKEY can lazy-expire arbitrary keys (similar behavior to active-expire), and put DELs in a transaction. Fix: When these commands are called without a parent exec-unit (e.g. not in EVAL or MULTI) we avoid wrapping their DELs in a transaction (for the same reasons active-expire and eviction avoids a transaction) This PR adds a per-command flag that indicates that the command may touch arbitrary keys (not the ones in the arguments), and uses that flag to avoid the MULTI-EXEC. For now, this flag is internal, since we're considering other solutions for the future. Note for cluster mode: if SCAN/RANDOMKEY is inside EVAL/MULTI it can still cause the same situation (as it always did), but it won't cause a CROSSSLOT because replicas and AOF do not perform slot checks. The problem with the above is mainly for 3rd party ecosystem tools that propagate commands from master to master, or feed an AOF file with redis-cli into a master. This PR aims to fix the regression in redis 7.0, and we opened #11792 to try to handle the bigger problem with lazy expire better for another release.
-
- 09 Feb, 2023 2 commits
-
-
Meir Shpilraien (Spielrein) authored
The PR adds support for the following flags on RedisModule_OpenKey: * REDISMODULE_OPEN_KEY_NONOTIFY - Don't trigger keyspace event on key misses. * REDISMODULE_OPEN_KEY_NOSTATS - Don't update keyspace hits/misses counters. * REDISMODULE_OPEN_KEY_NOEXPIRE - Avoid deleting lazy expired keys. * REDISMODULE_OPEN_KEY_NOEFFECTS - Avoid any effects from fetching the key In addition, added `RM_GetOpenKeyModesAll`, which returns the mask of all supported OpenKey modes. This allows the module to check, in runtime, which OpenKey modes are supported by the current Redis instance.
-
Binbin authored
Return an error when loadAppendOnlyFiles fails instead of exiting. DEBUF LOADAOF command is only meant to be used by the test suite, and only by tests that generated an AOF file first. So this change is ok (considering that the caller is likely to catch this error and die). This actually revert part of the code in #9012, and now DEBUG LOADAOF behaves the same as DEBUG RELOAD (returns an error when the load fails). Plus remove a `after 2000` in a test, which can save times (looks like copy paste error).
-
- 06 Feb, 2023 1 commit
-
-
filipe oliveira authored
If we have integer scores on the sorted set we're not using the fastest way to reply by calling `d2string` which uses `double2ll` and `ll2string` when it can, instead of `fpconv_dtoa`. This results by some 50% performance improvement in certain cases of integer scores for both RESP2 and RESP3, and no apparent impact on double scores. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 03 Feb, 2023 2 commits
-
-
Binbin authored
Failure happens in FreeBSD daily: ``` *** [err]: Test replication with parallel clients writing in different DBs in tests/integration/replication-4.tcl Expected [::redis::redisHandle2 dbsize] > 0 (context: type eval line 19 cmd {assert {[$master dbsize] > 0}} proc ::test) ``` The test is failing because db 9 has no data (default db), and according to the log, we can see that db 9 does not have a key: ``` ### Starting test Test replication with parallel clients writing in different DBs in tests/integration/replication-4.tcl 3338:S 03 Feb 2023 00:15:18.723 - DB 11: 1 keys (0 volatile) in 4 slots HT. 3338:S 03 Feb 2023 00:15:18.723 - DB 12: 141 keys (0 volatile) in 256 slots HT. ``` We use `wait_for_condition` to ensure that parallel clients have written data before calling stop_bg_complex_data. At the same time, `wait_for_condition` is also used to remove the above `after 1000`, which can save time in most cases.
-
Binbin authored
There is a timing issue in the new ACL log test: ``` *** [err]: ACL LOG aggregates similar errors together and assigns unique entry-id to new errors in tests/unit/acl.tcl Expected 1675382873989 < 1675382873989 (context: type eval line 15 cmd {assert {$timestamp_last_update_original < $timestamp_last_updated_after_update}} proc ::test) ``` Looking at the test code, we will check the `timestamp-last-updated` before and after a new ACL error occurs. Actually `WRONGPASS` errors can be executed very quickly on fast machines. For example, in the this case, the execution is completed within one millisecond. The error is easy to reproduce, if we reduce the number of the for loops, for example set to 2, and using --loop and --stop. Avoid this timing issue by adding an `after 1` before the new errors. The test was introduced in #11477.
-
- 02 Feb, 2023 2 commits
-
-
Roshan Khatri authored
Added 3 fields to the ACL LOG - adds entry_id, timestamp_created and timestamp_last_updated, which updates similar existing log error entries. The pair - entry_id, timestamp_created is a unique identifier of this entry, in case the node dies and is restarted, it can detect that if it's a new series. The primary use case of Unique id is to uniquely identify the error messages and not to detect if the server has restarted. entry-id is the sequence number of the entry (starting at 0) since the server process started. Can also be used to check if items were "lost" if they fell between periods. timestamp-created is the unix-time in ms at the time the entry was first created. timestamp-last-updated is the unix-time in ms at the time the entry was last updated Time_created gives the absolute time which better accounts for network time as compared to time since. It can also be older than 60 secs and presently there is no field that can display the original time of creation once the error entry is updated. The reason of timestamp_last_updated field is that it provides a more precise value for the “last time” an error was seen where as, presently it is only in the 60 second period. Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com>
-
Harkrishn Patro authored
Currently while a sharded pubsub message publish tries to propagate the message across the cluster, a NULL check is missing for clusterLink. clusterLink could be NULL if the link is causing memory beyond the set threshold cluster-link-sendbuf-limit and server terminates the link. This change introduces two things: Avoids the engine crashes on the publishing node if a message is tried to be sent to a node and the link is NULL. Adds a debugging tool CLUSTERLINK KILL to terminate the clusterLink between two nodes.
-
- 01 Feb, 2023 1 commit
-
-
Binbin authored
Test on x86 + TLS fail with this error: ``` *** [err]: Slave is able to detect timeout during handshake in tests/integration/replication.tcl Replica is not able to detect timeout ``` The replica logs is: ``` ### Starting test Slave is able to detect timeout during handshake in tests/integration/replication.tcl 7681:S 05 Jan 2023 00:21:56.635 * Non blocking connect for SYNC fired the event. 7681:S 05 Jan 2023 00:21:56.638 * Master replied to PING, replication can continue... 7681:S 05 Jan 2023 00:21:56.638 * Trying a partial resynchronization (request ef70638885500aad12dd673c68ca1541116a59fe:1). 7681:S 05 Jan 2023 00:22:56.894 # Failed to read response from the server: error:0A000126:SSL routines::unexpected eof while reading 7681:S 05 Jan 2023 00:22:56.894 # Master did not reply to PSYNC, will try later ``` This is another issue that appeared after #11640 was merged. This PR try to fix it. The idea is to make it stable in `wait_bgsave`, for example, it may wait until the next psync retry in the following situation: `Master did not reply to PSYNC, will try later` Other than that, the change will make the test more consistent / predictable since it'll mean the master is always frozen in the desired state (waiting for repl-diskless-sync-delay to happen, rather than earlier stages of the handshake).
-
- 31 Jan, 2023 1 commit
-
-
uriyage authored
In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation size in order to: > reduce the need for realloc calls by making the sds implicitly take over the internal fragmentation This change was done most sds functions, excluding `sdsRemoveFreeSpace` and `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer) we call sdsRemoveFreeSpace when we see excessive free space and want to trim it. so if we don't trim it exactly to size, the caller may still see excessive free space and call it again and again. However, this resulted in some excessive calls to realloc, even when there's no need and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13). It turns out that a call for realloc with jemalloc can be expensive even if it ends up doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid the call for realloc. in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common code. the difference between them was that sdsResize would avoid using SDS_TYPE_5, since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace would permit using SDS_TYPE_5 and get an optimal memory consumption. now both methods take a `would_regrow` argument that makes it more explicit. the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both. The new test that was added to cover this concern used to pass before this PR as well, this PR is just a performance optimization and cleanup. Benchmark: `redis-benchmark -c 100 -t set -d 512 -P 10 -n 100000000` on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs). some 4.5% improvement. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 16 Jan, 2023 3 commits
-
-
Oran Agra authored
Related to the hang reported in #11671 Currently, redis can disconnect a client due to reaching output buffer limit, it'll also avoid feeding that output buffer with more data, but it will keep running the loop in the command (despite the client already being marked for disconnection) This PR is an attempt to mitigate the problem, specifically for commands that are easy to abuse, specifically: KEYS, HRANDFIELD, SRANDMEMBER, ZRANDMEMBER. The RAND family of commands can take a negative COUNT argument (which is not bound to the number of elements in the key), so it's enough to create a key with one field, and then these commands can be used to hang redis. For KEYS the caller can use the existing keyspace in redis (if big enough).
-
Oran Agra authored
missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due to protocol limitations
-
Oran Agra authored
Authenticated users issuing specially crafted SETRANGE and SORT(_RO) commands can trigger an integer overflow, resulting with Redis attempting to allocate impossible amounts of memory and abort with an OOM panic.
-
- 08 Jan, 2023 1 commit
-
-
Gabi Ganam authored
Any value in the range of [0-1) turns to 0 when being cast from double to long long. This change rounds up instead of down for values that can't be stored precisely as long doubles.
-
- 04 Jan, 2023 2 commits
-
-
Oran Agra authored
TLDR: solve a problem introduced in Redis 7.0.6 (#11541) with RM_CommandFilterArgInsert being called from scripts, which can lead to memory corruption. Libc realloc can return the same pointer even if the size was changed. The code in freeLuaRedisArgv had an assumption that if the pointer didn't change, then the allocation didn't change, and the cache can still be reused. However, if rewriteClientCommandArgument or RM_CommandFilterArgInsert were used, it could be that we realloced the argv array, and the pointer didn't change, then a consecutive command being executed from Lua can use that argv cache reaching beyond its size. This was actually only possible with modules, since the decision to realloc was based on argc, rather than argv_len.
-
Oran Agra authored
Test on ARM + TLS often fail with this error: ``` *** [err]: Slave is able to detect timeout during handshake in tests/integration/replication.tcl Replica is not able to detect timeout ``` https://github.com/redis/redis-extra-ci/actions/runs/3727554226/jobs/6321797837 The replica logs show that in this case the replica got timeout before even getting a response to the PING command (instead of the SYNC command). it should have shown these: ``` * MASTER <-> REPLICA sync started * REPLICAOF 127.0.0.1:22112 enabled .... ### Starting test Slave enters handshake in tests/integration/replication.tcl * Non blocking connect for SYNC fired the event. ``` then: ``` * Master replied to PING, replication can continue... * Trying a partial resynchronization (request 50da9eff70d774f4e6cb723eb4b091440f215772:1). ``` and then hang for 5 seconds: ``` # Timeout connecting to the MASTER... * Reconnecting to MASTER 127.0.0.1:21112 after failure ``` but instead it got this (looks like it disconnected too early, and then tried to re-connect): ``` 10890:M 19 Dec 2022 01:32:54.794 * Ready to accept connections tls 10890:M 19 Dec 2022 01:32:54.809 - Accepted 127.0.0.1:41047 10890:M 19 Dec 2022 01:32:54.878 - Reading from client: error:0A000126:SSL routines::unexpected eof while reading 10890:M 19 Dec 2022 01:32:54.925 - Accepted 127.0.0.1:39207 10890:S 19 Dec 2022 01:32:55.463 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 10890:S 19 Dec 2022 01:32:55.463 * Connecting to MASTER 127.0.0.1:24126 10890:S 19 Dec 2022 01:32:55.463 * MASTER <-> REPLICA sync started 10890:S 19 Dec 2022 01:32:55.463 * REPLICAOF 127.0.0.1:24126 enabled (user request from 'id=4 addr=127.0.0.1:39207 laddr=127.0.0.1:24125 fd=8 name= age=1 idle=0 flags=N db=9 sub=0 psub=0 ssub=0 multi=-1 qbuf=43 qbuf-free=20431 argv-mem=21 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=22317 events=r cmd=slaveof user=default redir=-1 resp=2') ### Starting test Slave enters handshake in tests/integration/replication.tcl 10890:S 19 Dec 2022 01:32:55.476 * Non blocking connect for SYNC fired the event. 10890:S 19 Dec 2022 01:33:00.701 # Failed to read response from the server: (null) <- note this!! 10890:S 19 Dec 2022 01:33:00.701 # Master did not respond to command during SYNC handshake 10890:S 19 Dec 2022 01:33:01.002 * Connecting to MASTER 127.0.0.1:24126 10890:S 19 Dec 2022 01:33:01.002 * MASTER <-> REPLICA sync started ### Starting test Slave is able to detect timeout during handshake in tests/integration/replication.tcl 10890:S 19 Dec 2022 01:33:05.497 * Non blocking connect for SYNC fired the event. 10890:S 19 Dec 2022 01:33:05.500 * Master replied to PING, replication can continue... 10890:S 19 Dec 2022 01:33:05.510 * Trying a partial resynchronization (request 947e1956372a0e6c819cfec51c42cc7979b0c221:1). 10890:S 19 Dec 2022 01:34:05.833 # Failed to read response from the server: error:0A000126:SSL routines::unexpected eof while reading 10890:S 19 Dec 2022 01:34:05.833 # Master did not reply to PSYNC, will try later ``` This PR sets enables the 5 seconds timeout at a later stage to try and prevent the early disconnection.
-
- 01 Jan, 2023 1 commit
-
-
ranshid authored
*TL;DR* --------------------------------------- Following the discussion over the issue [#7551](https://github.com/redis/redis/issues/7551 ) We decided to refactor the client blocking code to eliminate some of the code duplications and to rebuild the infrastructure better for future key blocking cases. *In this PR* --------------------------------------- 1. reprocess the command once a client becomes unblocked on key (instead of running custom code for the unblocked path that's different than the one that would have run if blocking wasn't needed) 2. eliminate some (now) irrelevant code for handling unblocking lists/zsets/streams etc... 3. modify some tests to intercept the error in cases of error on reprocess after unblock (see details in the notes section below) 4. replace '$' on the client argv with current stream id. Since once we reprocess the stream XREAD we need to read from the last msg and not wait for new msg in order to prevent endless block loop. 5. Added statistics to the info "Clients" section to report the: * `total_blocking_keys` - number of blocking keys * `total_blocking_keys_on_nokey` - number of blocking keys which have at least 1 client which would like to be unblocked on when the key is deleted. 6. Avoid expiring unblocked key during unblock. Previously we used to lookup the unblocked key which might have been expired during the lookup. Now we lookup the key using NOTOUCH and NOEXPIRE to avoid deleting it at this point, so propagating commands in blocked.c is no longer needed. 7. deprecated command flags. We decided to remove the CMD_CALL_STATS and CMD_CALL_SLOWLOG and make an explicit verification in the call() function in order to decide if stats update should take place. This should simplify the logic and also mitigate existing issues: for example module calls which are triggered as part of AOF loading might still report stats even though they are called during AOF loading. *Behavior changes* --------------------------------------------------- 1. As this implementation prevents writing dedicated code handling unblocked streams/lists/zsets, since we now re-process the command once the client is unblocked some errors will be reported differently. The old implementation used to issue ``UNBLOCKED the stream key no longer exists`` in the following cases: - The stream key has been deleted (ie. calling DEL) - The stream and group existed but the key type was changed by overriding it (ie. with set command) - The key not longer exists after we swapdb with a db which does not contains this key - After swapdb when the new db has this key but with different type. In the new implementation the reported errors will be the same as if the command was processed after effect: **NOGROUP** - in case key no longer exists, or **WRONGTYPE** in case the key was overridden with a different type. 2. Reprocessing the command means that some checks will be reevaluated once the client is unblocked. For example, ACL rules might change since the command originally was executed and will fail once the client is unblocked. Another example is OOM condition checks which might enable the command to run and block but fail the command reprocess once the client is unblocked. 3. One of the changes in this PR is that no command stats are being updated once the command is blocked (all stats will be updated once the client is unblocked). This implies that when we have many clients blocked, users will no longer be able to get that information from the command stats. However the information can still be gathered from the client list. **Client blocking** --------------------------------------------------- the blocking on key will still be triggered the same way as it is done today. in order to block the current client on list of keys, the call to blockForKeys will still need to be made which will perform the same as it is today: * add the client to the list of blocked clients on each key * keep the key with a matching list node (position in the global blocking clients list for that key) in the client private blocking key dict. * flag the client with CLIENT_BLOCKED * update blocking statistics * register the client on the timeout table **Key Unblock** --------------------------------------------------- Unblocking a specific key will be triggered (same as today) by calling signalKeyAsReady. the implementation in that part will stay the same as today - adding the key to the global readyList. The reason to maintain the readyList (as apposed to iterating over all clients blocked on the specific key) is in order to keep the signal operation as short as possible, since it is called during the command processing. The main change is that instead of going through a dedicated code path that operates the blocked command we will just call processPendingCommandsAndResetClient. **ClientUnblock (keys)** --------------------------------------------------- 1. Unblocking clients on keys will be triggered after command is processed and during the beforeSleep 8. the general schema is: 9. For each key *k* in the readyList: ``` For each client *c* which is blocked on *k*: in case either: 1. *k* exists AND the *k* type matches the current client blocking type OR 2. *k* exists and *c* is blocked on module command OR 3. *k* does not exists and *c* was blocked with the flag unblock_on_deleted_key do: 1. remove the client from the list of clients blocked on this key 2. remove the blocking list node from the client blocking key dict 3. remove the client from the timeout list 10. queue the client on the unblocked_clients list 11. *NEW*: call processCommandAndResetClient(c); ``` *NOTE:* for module blocked clients we will still call the moduleUnblockClientByHandle which will queue the client for processing in moduleUnblockedClients list. **Process Unblocked clients** --------------------------------------------------- The process of all unblocked clients is done in the beforeSleep and no change is planned in that part. The general schema will be: For each client *c* in server.unblocked_clients: * remove client from the server.unblocked_clients * set back the client readHandler * continue processing the pending command and input buffer. *Some notes regarding the new implementation* --------------------------------------------------- 1. Although it was proposed, it is currently difficult to remove the read handler from the client while it is blocked. The reason is that a blocked client should be unblocked when it is disconnected, or we might consume data into void. 2. While this PR mainly keep the current blocking logic as-is, there might be some future additions to the infrastructure that we would like to have: - allow non-preemptive blocking of client - sometimes we can think that a new kind of blocking can be expected to not be preempt. for example lets imagine we hold some keys on disk and when a command needs to process them it will block until the keys are uploaded. in this case we will want the client to not disconnect or be unblocked until the process is completed (remove the client read handler, prevent client timeout, disable unblock via debug command etc...). - allow generic blocking based on command declared keys - we might want to add a hook before command processing to check if any of the declared keys require the command to block. this way it would be easier to add new kinds of key-based blocking mechanisms. Co-authored-by:
Oran Agra <oran@redislabs.com> Signed-off-by:
Ran Shidlansik <ranshid@amazon.com>
-
- 23 Dec, 2022 1 commit
-
-
Madelyn Olson authored
Attempt to harden cluster init-tests by doing two things: * Retry up to 3 times to join the cluster. Cluster meet is entirely idempotent, so it should stabilize if we missed a node. * Validate the connection is actually established, not just exists in the cluster list. Nodes can exist in handshake, but might later get dropped.
-
- 22 Dec, 2022 2 commits
-
-
Binbin authored
This test sometimes fails: ``` *** [err]: PSYNC2: Partial resync after Master restart using RDB aux fields with expire in tests/integration/psync2-master-restart.tcl Expected [status ::redis::redisHandle24 sync_partial_ok] == 1 (context: type eval line 49 cmd {assert {[status $replica sync_partial_ok] == 1}} proc ::test) ``` This is because the default repl-timeout value is 10s, sometimes the test got timeout, then it will do a reconnect, it will incr the sync_partial_ok counter, and then cause the test to fail.In this fix, we set the repl-timeout to a very large number to make sure we won't get the timeout.
-
Binbin authored
This test failed in FreeBSD: ``` *** [err]: PTTL returns time to live in milliseconds in tests/unit/expire.tcl Expected 836 > 900 && 836 <= 1000 (context: type eval line 5 cmd {assert {$ttl > 900 && $ttl <= 1000}} proc ::test) ``` On some slow machines, sometimes the test take close to 200ms to finish. We only set aside 100ms, so that caused the failure. Since the failure was around 800, change the condition to be >500.
-
- 20 Dec, 2022 1 commit
-
-
guybe7 authored
1. Get rid of server.core_propagates - we can just rely on module/call nesting levels 2. Rename in_nested_call to execution_nesting and update the comment 3. Remove module_ctx_nesting (redundant, we can use execution_nesting) 4. Modify postExecutionUnitOperations according to the comment (The main purpose of this PR) 5. trackingHandlePendingKeyInvalidations: Check the nesting level inside this function
-
- 18 Dec, 2022 2 commits
-
-
Oran Agra authored
I've seen the `BRPOPLPUSH with multiple blocked clients` test hang. this probably happened because rd2 blocked before rd1 and then it was also released first, and rd1 remained blocked. ``` r del blist{t} target1{t} target2{t} r set target1{t} nolist $rd1 brpoplpush blist{t} target1{t} 0 $rd2 brpoplpush blist{t} target2{t} 0 r lpush blist{t} foo assert_error "WRONGTYPE*" {$rd1 read} assert_equal {foo} [$rd2 read] assert_equal {foo} [r lrange target2{t} 0 -1] ``` changes: * added all missing calls for wait_for_blocked_client after issuing blocking commands) * removed some excessive `after 100` * fix undetected crossslot error in BRPOPLPUSH test * rollback changes to proto-max-bulk-len so external tests can be rerun
-
Oran Agra authored
Fix a flaky test that probably fails on overload timing issues. This unit starts with ``` # Set a threshold high enough to avoid spurious latency events. r config set latency-monitor-threshold 200 ``` but later the test measuring expire event changes the threshold. this fix is to revert it to 200 after that test. Got this error (ARM+TLS) ``` *** [err]: LATENCY RESET is able to reset events in tests/unit/latency-monitor.tcl Expected [r latency latest] eq {} (context: type eval line 3 cmd {assert {[r latency latest] eq {}}} proc ::test) ```
-
- 15 Dec, 2022 1 commit
-
-
filipe oliveira authored
Fixes a regression introduced by #11552 in 7.0.6. it causes replies in the GEO commands to contain garbage when the result is a very small distance (less than 1) Includes test to confirm indeed with junk in buffer now we properly reply
-
- 13 Dec, 2022 1 commit
-
-
Binbin authored
There is a race in the test: ``` *** [err]: Diskless load swapdb (async_loading): new database is exposed after swapping in tests/integration/replication.tcl Expected 'myvalue' to be equal to '' (context: type eval line 3 cmd {assert_equal [$replica GET mykey] ""} proc ::test) ``` When doing `$replica GET mykey`, the replica is using the old database. The reason may be that when doing `master client kill type replica`, the replica did not yet realize it got disconnected from the master. So the check of master_link_status fails, and the replica did not finish the swapdb and the loading. In that case, i think the solution is to check the sync_full stat on the master and wait for it to get incremented from the previous value. i.e. the way to know that we're done with the full sync is not to check that our state is up (could be up if we check too early), but rather check that the sync_full counter got incremented. During the reviewing, we found another race, in Aborted testType, the `$master config set rdb-key-save-delay 10000` is done after we already initiated the disconnection, so there's a chance that the replica will attempt to reconnect before that call, in which case if we fork() before it, the config will not take effect. Move it to above the disconnection. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 12 Dec, 2022 2 commits
-
-
Oran Agra authored
attach_to_replication_stream already stops pings, but it stops them on the server we connect to, and in this case it's a replica, and we need to stop them on the real master.
-
Binbin authored
There is a timing issue in the test, happens with valgrind: ``` *** [err]: diskless fast replicas drop during rdb pipe in tests/integration/replication.tcl log message of '"*Loading DB in memory*"' not found in ./tests/tmp/server.3580.246/stdout after line: 0 till line: 39 ``` The server logs: ``` 43465:S 03 Dec 2022 01:26:25.664 * Trying a partial resynchronization (request 15155fa24af0539b70428f9b41f4f7129d774560:1). 43465:S 03 Dec 2022 01:26:35.133 * Full resync from master: 8ddf5a3f7c8ca1061c6b29aa84e7c985c5b29c61:680 ``` From the logs, we can see it took almost 10s to get full resync response, happens with valgrind. it's extremely slow. So i guess it's just an insufficient wait_for_condition timeout. Set the time to 15s, and modify other similar places at the same time.
-
- 11 Dec, 2022 1 commit
-
-
Oran Agra authored
Clang Address Sanitizer tests started reporting unknown-crash on these tests due to the memcheck, disable the memcheck to avoid that noise.
-
- 09 Dec, 2022 2 commits
-
-
Binbin authored
In #11290, we added listpack encoding for SET object. But forgot to support it in zuiFind, causes ZINTER, ZINTERSTORE, ZINTERCARD, ZIDFF, ZDIFFSTORE to crash. And forgot to support it in RM_ScanKey, causes it hang. This PR add support SET listpack in zuiFind, and in RM_ScanKey. And add tests for related commands to cover this case. Other changes: - There is no reason for zuiFind to go into the internals of the SET. It can simply use setTypeIsMember and don't care about encoding. - Remove the `#include "intset.h"` from server.h reduce the chance of accidental intset API use. - Move setTypeAddAux, setTypeRemoveAux and setTypeIsMemberAux interfaces to the header. - In scanGenericCommand, use setTypeInitIterator and setTypeNext to handle OBJ_SET scan. - In RM_ScanKey, improve hash scan mode, use lpGetValue like zset, they can share code and better performance. The zuiFind part fixes #11578 Co-autho...
-
Oran Agra authored
We do defrag during AOF loading, but aim to detect fragmentation only once a second, so this test aims to slow down the AOF loading and mimic loading of a large file. On fast machines the sleep, plus the actual work we did was insufficient making it sleep longer so the test won't fail. The error we used to get is this one: Expected 0 > 100000 (context: type eval line 106 cmd {assert {$hits > 100000}} proc ::test)
-
- 08 Dec, 2022 1 commit
-
-
Binbin authored
From https://en.wikipedia.org/wiki/NaN#Display, it says that apart from nan and -nan, we can also get NAN and even nan(char-sequence) from libc. In #11482, our conclusion was that we wanna normalize it in Redis to a single nan type, like we already normalized inf. For this, we also reverted the assert_match part of the test added in #11506, using assert_equal to validate the changes.
-
- 07 Dec, 2022 1 commit
-
-
Harkrishn Patro authored
## Issue During the client input/output buffer processing, the memory usage is incrementally updated to keep track of clients going beyond a certain threshold `maxmemory-clients` to be evicted. However, this additional tracking activity leads to unnecessary CPU cycles wasted when no client-eviction is required. It is applicable in two cases. * `maxmemory-clients` is set to `0` which equates to no client eviction (applicable to all clients) * `CLIENT NO-EVICT` flag is set to `ON` which equates to a particular client not applicable for eviction. ## Solution * Disable client memory usage tracking during the read/write flow when `maxmemory-clients` is set to `0` or `client no-evict` is `on`. The memory usage is tracked only during the `clientCron` i.e. it gets periodically updated. * Cleanup the clients from the memory usage bucket when client eviction is disabled. * When the maxmemory-clients config is enabled or disabled at runtime, we immediately update the memory usage buckets for all clients (tested scanning 80000 took some 20ms) Benchmark shown that this can improve performance by about 5% in certain situations. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 05 Dec, 2022 1 commit
-
-
filipe oliveira authored
This mechanism aims to reduce calls to malloc and free when preparing the arguments the script sends to redis commands. This is a mechanism was originally implemented in 48c49c48 and 4f686555 , and was removed in #10220 (thinking it's not needed and that it has no impact), but it now turns out it was wrong, and it indeed provides some 5% performance improvement. The implementation is a little bit too simplistic, it assumes consecutive calls use the same size in the same arg index, but that's arguably sufficient since it's only aimed at caching very small things. We could even consider always pre-allocating args to the full LUA_CMD_OBJCACHE_MAX_LEN (64 bytes) rather than the right size for the argument, that would increase the chance they'll be able to be re-used. But in some way this is already happening since we're using sdsalloc, which in turn uses s_malloc_usable and takes ownership of the full side of the allocation, so we are padded to the allocator bucket size. Co-authored-by:
Oran Agra <oran@redislabs.com> Co-authored-by:
sundb <sundbcn@gmail.com>
-
- 04 Dec, 2022 1 commit
-
-
filipe oliveira authored
GEODIST used snprintf("%.4f") for the reply using addReplyDoubleDistance, which was slow. This PR optimizes it without breaking compatibility by following the approach of ll2string with some changes to match the use case of distance and precision. I.e. we multiply it by 10000 format it as an integer, and then add a decimal point. This can achieve about 35% increase in the achievable ops/sec. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 01 Dec, 2022 1 commit
-
-
Binbin authored
The test failed with ERR DUMP payload version or checksum are wrong. And it only fails on CentOS, this is due to the fact that tcl8.5 does not correctly parse the hexadecimal abbreviation. And in Ubuntu we are using tcl8.6.
-
- 30 Nov, 2022 3 commits
-
-
Oran Agra authored
This test sets the master ping interval to 1 hour, in order to avoid pings in the replicatoin stream incrementing the replication offset, however, it didn't increase the repl-timeout so on slow machines where the test took more than 60 seconds, the replicas would drop and reconnect. ``` *** [err]: PSYNC2: Partial resync after restart using RDB aux fields in tests/integration/psync2.tcl Replica didn't partial sync ``` The test would detect 4 additional partial syncs where it expects only one.
-
guybe7 authored
1. "Fixed" the current code so that seen-time/idle actually refers to interaction attempts (as documented; breaking change) 2. Added active-time/inactive to refer to successful interaction (what seen-time/idle used to be) At first, I tried to avoid changing the behavior of seen-time/idle but then realized that, in this case, the odds are the people read the docs and implemented their code based on the docs (which didn't match the behavior). For the most part, that would work fine, except that issue #9996 was found. I was working under the assumption that people relied on the docs, and for the most part, it could have worked well enough. so instead of fixing the docs, as I would usually do, I fixed the code to match the docs in this particular case. Note that, in case the consumer has never read any entries, the values for both "active-time" (XINFO FULL) and "inactive" (XINFO CONSUMERS) will be -1, meaning here that the consumer was never active. Note that seen/active time is only affected by XREADGROUP / X[AUTO]CLAIM, not by XPENDING, XINFO, and other "read-only" stream CG commands (always has been, even before this PR) Other changes: * Another behavioral change (arguably a bugfix) is that XREADGROUP and X[AUTO]CLAIM create the consumer regardless of whether it was able to perform some reading/claiming * RDB format change to save the `active_time`, and set it to the same value of `seen_time` in old rdb files.
-
Huang Zhw authored
Add a new module event `RedisModule_Event_Key`, this event is fired when a key is removed from the keyspace. The event includes an open key that can be used for reading the key before it is removed. Modules can also extract the key-name, and use RM_Open or RM_Call to access key from within that event, but shouldn't modify anything from within this event. The following sub events are available: - `REDISMODULE_SUBEVENT_KEY_DELETED` - `REDISMODULE_SUBEVENT_KEY_EXPIRED` - `REDISMODULE_SUBEVENT_KEY_EVICTED` - `REDISMODULE_SUBEVENT_KEY_OVERWRITE` The data pointer can be casted to a RedisModuleKeyInfo structure with the following fields: ``` RedisModuleKey *key; // Opened Key ``` ### internals * We also add two dict functions: `dictTwoPhaseUnlinkFind` finds an element from the table, also get the plink of the entry. The entry is returned if the element is found. The user should later call `dictTwoPhaseUnlinkFree` with it in order to unlink and release it. Otherwise if the key is not found, NULL is returned. These two functions should be used in pair. `dictTwoPhaseUnlinkFind` pauses rehash and `dictTwoPhaseUnlinkFree` resumes rehash. * We change `dbOverwrite` to `dbReplaceValue` which just replaces the value of the key and doesn't fire any events. The "overwrite" part (which emits events) is just when called from `setKey`, the other places that called dbOverwrite were ones that just update the value in-place (INCR*, SPOP, and dbUnshareStringValue). This should not have any real impact since `moduleNotifyKeyUnlink` and `signalDeletedKeyAsReady` wouldn't have mattered in these cases anyway (i.e. module keys and stream keys didn't have direct calls to dbOverwrite) * since we allow doing RM_OpenKey from withing these callbacks, we temporarily disable lazy expiry. * We also temporarily disable lazy expiry when we are in unlink/unlink2 callback and keyspace notification callback. * Move special definitions to the top of redismodule.h This is needed to resolve compilation errors with RedisModuleKeyInfoV1 that carries a RedisModuleKey member. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 28 Nov, 2022 2 commits
-
-
Mingyi Kang authored
Hyperloglog avoid allocate more than 'server.hll_sparse_max_bytes' bytes of memory for sparse representation (#11438) Before this PR, we use sdsMakeRoomFor() to expand the size of hyperloglog string (sparse representation). And because sdsMakeRoomFor() uses a greedy strategy (allocate about twice what we need), the memory we allocated for the hyperloglog may be more than `server.hll_sparse_max_bytes` bytes. The memory more than` server.hll_sparse_max_bytes` will be wasted. In this pull request, tone down the greediness of the allocation growth, and also make sure it'll never request more than `server.hll_sparse_max_bytes`. This could in theory mean the size of the hyperloglog string is insufficient for the increment we need, should be ok since in this case we promote the hyperloglog to dense representation, an assertion was added to make sure. This PR also add some tests and fixes some typo and indentation issues.
-
Binbin authored
In replica, the key expired before master's `INCR` was arrived, so INCR creates a new key in the replica and the test failed. ``` *** [err]: Replication of an expired key does not delete the expired key in tests/integration/replication-4.tcl Expected '0' to be equal to '1' (context: type eval line 13 cmd {assert_equal 0 [$slave exists k]} proc ::test) ``` This test is very likely to do a false positive if the `wait_for_ofs_sync` takes longer than the expiration time, so give it a few more chances. The test was introduced in #9572.
-