- 21 Nov, 2024 2 commits
-
-
Oran Agra authored
To complement the work done in #13133. it added the script VMs memory to be counted as part of zmalloc, but that means they should be also counted as part of the non-value overhead. this commit contains some refactoring to make variable names and function names less confusing. it also adds a new field named `script.VMs` into the `MEMORY STATS` command. additionally, clear scripts and stats between tests in external mode (which is related to how this issue was discovered)
-
nafraf authored
Fix to https://github.com/redis/redis/issues/13650 providing an invalid config to a module with datatype crashes when redis tries to unload the module due to the invalid config --------- Co-authored-by:
debing.sun <debing.sun@redis.com>
-
- 14 Nov, 2024 1 commit
-
-
debing.sun authored
inspred by https://github.com/redis/redis/pull/12730 Before this PR, we allocate new memory to store the user command arguments, however, if the size of the current `c->argv` is larger than the current command, we can reuse the previously allocated argv to avoid allocating new memory for the current command. And we will free `c->argv` in client cron when the client is idle for 2 seconds. --------- Co-authored-by:
Ozan Tezcan <ozantezcan@gmail.com>
-
- 12 Nov, 2024 1 commit
-
-
Moti Cohen authored
Improve the performance of crc64 for large batches by processing large number of bytes in parallel and combining the results. --------- Co-authored-by:
Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by:
Madelyn Olson <madelyneolson@gmail.com> Co-authored-by:
Josiah Carlson <josiah.carlson@gmail.com>
-
- 11 Nov, 2024 1 commit
-
-
Ozan Tezcan authored
If `hide-user-data-from-log` config is enabled, we don't print client argv in the crashlog to avoid leaking user info. Though, debugging a crash becomes harder as we don't see the command arguments causing the crash. With this PR, we'll be printing command tokens to the log. As we have command tokens defined in json schema for each command, using this data, we can find tokens in the client argv. e.g. `SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the log. Modules should introduce their command structure via `RM_SetCommandInfo()`. Then, on a crash we'll able to know module command tokens.
-
- 08 Nov, 2024 1 commit
-
-
Nugine authored
This PR optimizes the performance of HyperLogLog commands (PFCOUNT, PFMERGE) by adding AVX2 fast paths. Two AVX2 functions are added for conversion between raw representation and dense representation. They are 15 ~ 30 times faster than scalar implementaion. Note that sparse representation is not accelerated. AVX2 fast paths are enabled when the CPU supports AVX2 (checked at runtime) and the hyperloglog configuration is default (HLL_REGISTERS == 16384 && HLL_BITS == 6). When merging 3 dense hll structures, the benchmark shows a 12x speedup compared to the scalar version. ``` pfcount key1 key2 key3 pfmerge keyall key1 key2 key3 ``` ``` ====================================================================================================== Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec ------------------------------------------------------------------------------------------------------ PFCOUNT-scalar 5570.09 35.89060 32.51100 65.27900 69.11900 299.17 PFCOUNT-avx2 72604.92 2.82072 2.73500 5.50300 7.13500 3899.68 ------------------------------------------------------------------------------------------------------ PFMERGE-scalar 7879.13 25.52156 24.19100 46.33500 48.38300 492.45 PFMERGE-avx2 126448.64 1.58120 1.53500 3.08700 4.89500 7903.04 ------------------------------------------------------------------------------------------------------ scalar: redis:unstable 9906daf5 avx2: Nugine:hll-simd 02e09f85ac07eace50ebdddd0fd70822f7b9152d CPU: 13th Gen Intel® Core™ i9-13900H × 20 Memory: 32.0 GiB OS: Ubuntu 22.04.5 LTS ``` Experiment repo: https://github.com/Nugine/redis-hyperloglog Benchmark script: https://github.com/Nugine/redis-hyperloglog/blob/main/scripts/memtier.sh Algorithm: https://github.com/Nugine/redis-hyperloglog/blob/main/cpp/bench.cpp resolves #13551 --------- Co-authored-by:
Yuan Wang <wangyuancode@163.com> Co-authored-by:
debing.sun <debing.sun@redis.com>
-
- 04 Nov, 2024 1 commit
-
-
David Dougherty authored
This PR replaces old .../topics/... links with current links, specifically for the modules-api-ref.md file and the new automation that Paolo Lazzari is working on. A few of the topics links have redirects, but some don't. Best to use updated links.
-
- 30 Oct, 2024 3 commits
-
-
guybe7 authored
Added a log of the keyname in the test modules to reproduce the problem (tests crash without the fix)
-
Moti Cohen authored
On RDB load error, if an invalid `expireAt` value is read, `dupSearchDict` is not released.
-
debing.sun authored
`lpCompare()` in `quicklistCompare()` will call `lpGet()` again, which would be a waste. The change will result in a boost for all commands that use `quicklistCompre()`, including `linsert`, `lpos` and `lrem`.
-
- 29 Oct, 2024 1 commit
-
-
Moti Cohen authored
This PR adds a new section to the `INFO` command output, called `keysizes`. This section provides detailed statistics on the distribution of key sizes for each data type (strings, lists, sets, hashes and zsets) within the dataset. The distribution is tracked using a base-2 logarithmic histogram. # Motivation Currently, Redis lacks a built-in feature to track key sizes and item sizes per data type at a granular level. Understanding the distribution of key sizes is critical for monitoring memory usage and optimizing performance, particularly in large datasets. This enhancement will allow users to inspect the size distribution of keys directly from the `INFO` command, assisting with performance analysis and capacity planning. # Changes New Section in `INFO` Command: A new section called `keysizes` has been added to the `INFO` command output. This section reports a per-database, per-type histogram of key sizes. It provides insights into how many keys fall into specific size ranges (represented in powers of 2). **Example output:** ``` 127.0.0.1:6379> INFO keysizes # Keysizes db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2 db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42 db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89, db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1 ``` ## Future Use Cases: The key size distribution is collected per slot as well, laying the groundwork for future enhancements related to Redis Cluster.
-
- 28 Oct, 2024 1 commit
-
-
Shockingly Good authored
The crash happens whenever the user isn't accessible, for example, it isn't set for the context (when it is temporary) or in some other cases like `notifyKeyspaceEvent`. To properly check for the ACL compliance, we need to get the user name and the user to invoke other APIs. However, it is not possible if it crashes, and it is impossible to work that around in the code since we don't know (and **shouldn't know**!) when it is available and when it is not.
-
- 22 Oct, 2024 1 commit
-
-
opt-m authored
From 7.4, Redis allows `GET` options in cluster mode when the pattern maps to the same slot as the key, but GET # pattern that represents key itself is missed. This commit resolves it, bug report #13607. --------- Co-authored-by:
Yuan Wang <yuan.wang@redis.com>
-
- 18 Oct, 2024 1 commit
-
-
debing.sun authored
introduced by https://github.com/redis/redis/pull/13359 failure CI on ARM64: https://github.com/redis/redis-extra-ci/actions/runs/11377893230/job/31652773710 --------- Co-authored-by:
Ozan Tezcan <ozantezcan@gmail.com> Co-authored-by:
ShooterIT <wangyuancode@163.com>
-
- 17 Oct, 2024 2 commits
-
-
hanhui365 authored
Nowadays popcnt instruction is almost supported by X86 machine, which is used to calculate "Hamming weight", it can bring much performance boost in redis bitcount comand. --------- Signed-off-by: hanhui365(hanhui@hygon.cn) Co-authored-by:
debing.sun <debing.sun@redis.com> Co-authored-by:
oranagra <oran@redislabs.com> Co-authored-by:
Nugine <nugine@foxmail.com>
-
Yuan Wang authored
After running test in local, there will be a file named `.rediscli_history_test`, and it is not in `.gitignore` file, so this is considered to have changed the code base. It is a little annoying, this commit just clean up the temporary file. We should delete `.rediscli_history_test` in the end since the second server tests also write somethings into it, to make it corresponding, i put `set ::env(REDISCLI_HISTFILE) ".rediscli_history_test"` at the beginning. Maybe we also can add this file into `.gitignore`
-
- 15 Oct, 2024 3 commits
-
-
YaacovHazan authored
Update redisbloom, redisjson and redistimeseries versions to 7.99.1 Co-authored-by:
YaacovHazan <yaacov.hazan@redislabs.com>
-
paoloredis authored
We only want to trigger the workflow on the documentation repository for the latest release
-
YaacovHazan authored
- Add a new 'EXPERIMENTAL' command flag, which causes the command generator to skip over it and make the command to be unavailable for execution - Skip experimental tests by default - Move the SFLUSH tests from the old framework to the new one --------- Co-authored-by:
YaacovHazan <yaacov.hazan@redislabs.com>
-
- 12 Oct, 2024 1 commit
-
- 10 Oct, 2024 1 commit
-
-
guybe7 authored
1. `dbRandomKey`: excessive call to `dbFindExpires` (will always return 1 if `allvolatile` + anyway called inside `expireIfNeeded` 2. Add `deleteKeyAndPropagate` that is used by both expiry/eviction 3. Change the order of calls in `expireIfNeeded` to save redundant calls to `keyIsExpired` 4. `expireIfNeeded`: move `OBJ_STATIC_REFCOUNT` to `deleteKeyAndPropagate` 5. `performEvictions` now uses `deleteEvictedKeyAndPropagate` 6. active-expire: moved `postExecutionUnitOperations` inside `activeExpireCycleTryExpire` 7. `activeExpireCycleTryExpire`: less indentation + expire a key if `now == t` 8. rename `lazy_expire_disabled` to `allow_access_expired`
-
- 08 Oct, 2024 5 commits
-
-
Oran Agra authored
-
Oran Agra authored
The '%' rule must contain one or both of R/W
-
Oran Agra authored
INT_MIN value must be explicitly checked, and cannot be negated.
-
alonre24 authored
Update search target path and version from M02
-
chx9 authored
fix typo in test_helper.tcl: even driven => event driven
-
- 29 Sep, 2024 1 commit
-
-
Moti Cohen authored
This PR introduces a new `SFLUSH` command to cluster mode that allows partial flushing of nodes based on specified slot ranges. Current implementation is designed to flush all slots of a shard, but future extensions could allow for more granular flushing. **Command Usage:** `SFLUSH <start-slot> <end-slot> [<start-slot> <end-slot>]* [SYNC|ASYNC]` This command removes all data from the specified slots, either synchronously or asynchronously depending on the optional SYNC/ASYNC argument. **Functionality:** Current imp of `SFLUSH` command verifies that the provided slot ranges are valid and cover all of the node's slots before proceeding. If slots are partially or incorrectly specified, the command will fail and return an error, ensuring that all slots of a node must be fully covered for the flush to proceed. The function supports both synchronous (default) and asynchronous flushing. In addition, if possible, SFLUSH SYNC will be run as blocking ASYNC as an optimization.
-
- 25 Sep, 2024 2 commits
-
-
Ozan Tezcan authored
This PR is based on https://github.com/valkey-io/valkey/pull/996 Currently, for operations like SUNION or SDIFF, temporary set object can be intset or listpack. Search operation is costly for these encodings. This patch tries to set the temporary set object as hash table by default. It also tries to determine correct encoding for the temporary set object to reduce the unnecessary conversation. This change is supposed to give performance boost for tests like: - [memtier_benchmark-2keys-set-10-100-elements-sdiff](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sdiff.yml) 66.2% IMPROVEMENT - [memtier_benchmark-2keys-set-10-100-elements-sunion](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sunion.yml ) 126.5% IMPROVEMENT ------- Co-authored-by:
Lipeng Zhu <lipeng.zhu@intel.com> Co-authored-by:
Wangyang Guo <wangyang.guo@intel.com> Co-authored-by:
Lipeng Zhu <lipeng.zhu@intel.com> Co-authored-by:
Wangyang Guo <wangyang.guo@intel.com>
-
Moti Cohen authored
This PR is based on valkey-io/valkey#829 Previously, ZUNION and ZUNIONSTORE commands used a temporary accumulator dict and at the end copied it as-is to dstzset->dict. This PR removes accumulator and directly stores into dstzset->dict, eliminating the extra copy. Co-authored-by: Rayacoo zisong.cw@alibaba-inc.com
-
- 23 Sep, 2024 2 commits
-
-
Moti Cohen authored
Test 1 - give more time for expiration Test 2 - Evaluate expiration time boundaries [+1,+2] before setting expiration [+1] Test 3 - Avoid race on test HFEs propagated to replica
-
debing.sun authored
#13258 Incorrect use of free instead of zfree
-
- 19 Sep, 2024 1 commit
-
-
Moti Cohen authored
The PR extends `RedisModule_OpenKey`'s flags to include `REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired keys. It also allows to access expired subkeys. Currently relevant only for hash fields and has its impact on `RM_HashGet` and `RM_Scan`.
-
- 18 Sep, 2024 1 commit
-
-
debing.sun authored
Since `\\` is only one character, we need to add an extra space to the right.
-
- 15 Sep, 2024 3 commits
-
-
adamiBs authored
This PR introduces the installation of the `musl`-based version of Rust, in order to support alpine-based runtime environments (Rust is used by [RedisJSON](https://github.com/RedisJSON/RedisJSON)).
-
Filipe Oliveira (Redis) authored
Replace usage of _addReplyLongLongWithPrefix with specific bulk/mbulk functions to reduce condition checks in hotpath. (#13520) Instead of adding runtime logic to decide which prefix/shared object to use when doing the reply we can simply use an inline method to avoid runtime overhead of condition checks, and also keep the code change small. Preliminary data show improvements on commands that heavily rely on bulk/mbulk replies (example of LRANGE). --------- Co-authored-by:
debing.sun <debing.sun@redis.com>
-
Filipe Oliveira (Personal) authored
Fixes #8825 We're using the fast_float library[1] in our (compiled-in) floating-point fast_float_strtod implementation for faster and more portable parsing of 64 decimal strings. The single file fast_float.h is an amalgamation of the entire library, which can be (re)generated with the amalgamate.py script (from the fast_float repository) via the command: ``` python3 ./script/amalgamate.py --license=MIT > $REDIS_SRC/deps/fast_float/fast_float.h ``` [1]: https://github.com/fastfloat/fast_float The used commit from fast_float library was the one from https://github.com/fastfloat/fast_float/releases/tag/v3.10.1 --------- Co-authored-by:
fcostaoliveira <filipe@redis.com>
-
- 13 Sep, 2024 2 commits
-
-
Filipe Oliveira (Redis) authored
Optimize HSCAN/ZSCAN command in case of listpack encoding: avoid the usage of intermediate list (#13531) Similar to #13530 , applied to HSCAN and ZSCAN in case of listpack encoding. **Preliminary benchmark results showcase an improvement of 108% on the achievable ops/sec for ZSCAN and 65% for HSCAN**. --------- Co-authored-by:
debing.sun <debing.sun@redis.com>
-
debing.sun authored
in https://github.com/redis/redis/pull/13519, when `eb` is empty, `isRax` is not correctly initialized to 0, which can lead to `ebStop()` potentially entering the wrong rax branch.
-
- 12 Sep, 2024 2 commits
-
-
Filipe Oliveira (Redis) authored
Optimize SSCAN command in case of listpack or intset encoding: avoid the usage of intermediate list. From 2N to N iterations (#13530) On SSCAN, in case of listpack and intset encoding we actually reply the entire set, and always reply with the cursor 0. For those cases, we don't need to accumulate the replies in a list and can completely avoid the overhead of list appending and then iterating over the list again -- meaning we do N iterations instead of 2N iterations over the SET and save intermediate memory as well. Preliminary benchmarks, `SSCAN set:100 0`, showcased an improvement of 60% as visible bellow on a SET with 100 string elements (listpack encoded).
-
Moti Cohen authored
Add basic iterator API for ebuckets of start, next, nextBucket and stop.
-