- 15 Sep, 2024 1 commit
-
-
Filipe Oliveira (Personal) authored
Fixes #8825 We're using the fast_float library[1] in our (compiled-in) floating-point fast_float_strtod implementation for faster and more portable parsing of 64 decimal strings. The single file fast_float.h is an amalgamation of the entire library, which can be (re)generated with the amalgamate.py script (from the fast_float repository) via the command: ``` python3 ./script/amalgamate.py --license=MIT > $REDIS_SRC/deps/fast_float/fast_float.h ``` [1]: https://github.com/fastfloat/fast_float The used commit from fast_float library was the one from https://github.com/fastfloat/fast_float/releases/tag/v3.10.1 --------- Co-authored-by:
fcostaoliveira <filipe@redis.com>
-
- 06 Aug, 2024 1 commit
-
-
Anurag Bandyopadhyay authored
Close #13316 --------- Co-authored-by:
Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com> Co-authored-by:
debing.sun <debing.sun@redis.com>
-
- 10 May, 2024 1 commit
-
-
ClaytonNorthey92 authored
added reverse history search to redis-cli, use it with the following: * CTRL+R : enable search backward mode, and search next one when pressing CTRL+R again until reach index 0. ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (reverse-i-search): # press CTRL+R (reverse-i-search): keys two # input `keys` (reverse-i-search): keys one # press CTRL+R again (reverse-i-search): keys one # press CTRL+R again, still `keys one` due to reaching index 0 (i-search): keys two # press CTRL+S, enable search forward (i-search): keys two # press CTRL+S, still `keys one` due to reaching index 1 ``` * CTRL+S : enable search forward mode, and search next one when pressing CTRL+S again until reach index 0. ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (i-search): # press CTRL+S (i-search): keys one # input `keys` (i-search): keys two # press CTRL+S again (i-search): keys two # press CTRL+R again, still `keys two` due to reaching index 0 (reverse-i-search): keys one # press CTRL+R, enable search backward (reverse-i-search): keys one # press CTRL+S, still `keys one` due to reaching index 1 ``` * CTRL+G : disable ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (reverse-i-search): # press CTRL+R (reverse-i-search): keys two # input `keys` 127.0.0.1:6379> # press CTRL+G ``` * CTRL+C : disable ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (reverse-i-search): # press CTRL+R (reverse-i-search): keys two # input `keys` 127.0.0.1:6379> # press CTRL+G ``` * TAB : use the current search result and exit search mode ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (reverse-i-search): # press CTRL+R (reverse-i-search): keys two # input `keys` 127.0.0.1:6379> keys two # press TAB ``` * ENTER : use the current search result and execute the command ``` 127.0.0.1:6379> keys one 127.0.0.1:6379> keys two (reverse-i-search): # press CTRL+R (reverse-i-search): keys two # input `keys` 127.0.0.1:6379> keys two # press ENTER (empty array) 127.0.0.1:6379> ``` * any arrow key will disable reverse search your result will have the search match bolded, you can press enter to execute the full result note: I have _only added this for multi-line mode_, as it seems to be forced that way when `repl` is called Closes: https://github.com/redis/redis/issues/8277 --------- Co-authored-by:
Clayton Northey <clayton@knowbl.com> Co-authored-by:
Viktor Söderqvist <viktor.soderqvist@est.tech> Co-authored-by:
debing.sun <debing.sun@redis.com> Co-authored-by:
Bjorn Svensson <bjorn.a.svensson@est.tech> Co-authored-by:
Viktor Söderqvist <viktor@zuiderkwast.se>
-
- 22 Feb, 2024 1 commit
-
-
debing.sun authored
Implement #12699 This PR exposing Lua os.clock() api for getting the elapsed time of Lua code execution. Using: ```lua local start = os.clock() ... do something ... local elpased = os.clock() - start ``` --------- Co-authored-by:
Meir Shpilraien (Spielrein) <meir@redis.com> Co-authored-by:
Madelyn Olson <34459052+madolson@users.noreply.github.com>
-
- 15 Dec, 2023 1 commit
-
-
Binbin authored
redis-cli avoids saving sensitive commands in it's history (doesn't persist them to the history file). this means that if you had a typo and you wanna re-run the command, you can't easily do that. This PR changes that to keep an in-memory history of all the redacted commands, and just not persist them to disk. This way we would be able to press the up arrow and re-try the command freely, and it'll just not survive a redis-cli restart.
-
- 05 Aug, 2023 1 commit
-
-
sundb authored
This PR mainly fixes a possible integer overflow in `json_append_string()`. When we use `cjson.encoding()` to encode a string larger than 2GB, at specific compilation flags, an integer overflow may occur leading to truncation, resulting in the part of the string larger than 2GB not being encoded. On the other hand, this overflow doesn't cause any read or write out-of-range or segment fault. 1) using -O0 for lua_cjson (`make LUA_DEBUG=yes`) In this case, `i` will overflow and leads to truncation. When `i` reaches `INT_MAX+1` and overflows to INT_MIN, when compared to len, `i` (1000000..00) is expanded to 64 bits signed integer (1111111.....000000) . At this point i will be greater than len and jump out of the loop, so `for (i = 0; i < len; i++)` will loop up to 2^31 times, and the part of larger than 2GB will be truncated. ```asm `i` => -0x24(%rbp) <+253>: addl $0x1,-0x24(%rbp) ; overflow if i large than 2^31 <+257>: mov -0x24(%rbp),%eax <+260>: movslq %eax,%rdx ; move a 32-bit value with sign extension into a 64-bit signed <+263>: mov -0x20(%rbp),%rax <+267>: cmp %rax,%rdx ; check `i < len` <+270>: jb 0x212600 <json_append_string+148> ``` 2) using -O2/-O3 for lua_cjson (`make LUA_DEBUG=no`, **the default**) In this case, because singed integer overflow is an undefined behavior, `i` will not overflow. `i` will be optimized by the compiler and use 64-bit registers for all subsequent instructions. ```asm <+180>: add $0x1,%rbx ; Using 64-bit register `rbx` for i++ <+184>: lea 0x1(%rdx),%rsi <+188>: mov %rsi,0x10(%rbp) <+192>: mov %al,(%rcx,%rdx,1) <+195>: cmp %rbx,(%rsp) ; check `i < len` <+199>: ja 0x20b63a <json_append_string+154> ``` 3) using 32bit Because `strbuf_ensure_empty_length()` preallocates memory of length (len * 6 + 2), in 32-bit `cjson.encode()` can only handle strings smaller than ((2 ^ 32) - 3 ) / 6. So 32bit is not affected. Also change `i` in `strbuf_append_string()` to `size_t`. Since its second argument `str` is taken from the `char2escape` string array which is never larger than 6, so `strbuf_append_string()` is not at risk of overflow (the bug was unreachable).
-
- 13 Jul, 2023 1 commit
-
-
Chayim I. Kirshen authored
-
- 10 Jul, 2023 1 commit
-
-
Oran Agra authored
* Fix integer overflows due to using wrong integer size. * Add assertions / panic when overflow still happens. * Deletion of dead code to avoid need to maintain it * Some changes are not because of bugs, but rather paranoia. * Improve cmsgpack and cjson test coverage. Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
- 18 Jun, 2023 1 commit
-
-
Oran Agra authored
Apparently for large size classes Jemalloc allocate some extra memory (can be up to 25% overhead for allocations of 16kb). see https://github.com/jemalloc/jemalloc/issues/1098#issuecomment-1589870476 p.s. from Redis's perspective that looks like external fragmentation, (i.e. allocated bytes will be low, and active pages bytes will be large) which can cause active-defrag to eat CPU cycles in vain. Some details about this mechanism we disable: --------------------------------------------------------------- Disabling this mechanism only affects large allocations (above 16kb) Not only that it isn't expected to cause any performance regressions, it's actually recommended, unless you have a specific workload pattern and hardware that benefit from this feature -- by default it's enabled and adds address randomization to all large buffers, by over allocating 1 page per large size class, and offsetting into that page to make the starting address of the user buffer randomized. Workloads such as scientific computation often handle multiple big matrixes at the same time, and the randomization makes sure that the cacheline level accesses don't suffer bad conflicts (when they all start from page-aligned addresses). However the downsize is also quite noticeable, like you observed that extra page per large size can cause memory overhead, plus the extra TLB entry. The other factor is, hardware in the last few years started doing the randomization at the hardware level, i.e. the address to cacheline mapping isn't a direct mapping anymore. So there's debate to disable the randomization by default, but we are still hesitant because when it matters, it could matter a lot, and having it enabled by default limits that worst case behavior, even though it means the majority of workloads suffers a regression. So in short, it's safe and offers better performance in most cases.
-
- 04 Jun, 2023 1 commit
-
-
Yossi Gottlieb authored
* We patch hiredis rather than rely on having a compatible sds version. * We now have better test coverage for redis-cli and redis-benchmark.
-
- 01 May, 2023 1 commit
-
-
Oran Agra authored
* Regenerate configure script sccording to deps/README * update iget_defrag_hint by following changes to arena_dalloc_no_tcache
-
- 18 Apr, 2023 1 commit
-
-
sundb authored
This PR is to fix the compilation warnings and errors generated by the latest complier toolchain, and to add a new runner of the latest toolchain for daily CI. ## Fix various compilation warnings and errors 1) jemalloc.c COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` src/jemalloc.c:1028:7: warning: suspicious concatenation of string literals in an array initialization; did you mean to separate the elements with a comma? [-Wstring-concatenation] "/etc/malloc.conf", ^ src/jemalloc.c:1027:3: note: place parentheses around the string literal to silence warning "\"name\" of the file referenced by the symbolic link named " ^ ``` REASON: the compiler to alert developers to potential issues with string concatenation that may miss a comma, just like #9534 which misses a comma. SOLUTION: use `()` to tell the compiler that these two line strings are continuous. 2) config.h COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` In file included from quicklist.c:36: ./config.h:319:76: warning: attribute declaration must precede definition [-Wignored-attributes] char *strcat(char *restrict dest, const char *restrict src) __attribute__((deprecated("please avoid use of unsafe C functions. prefer use of redis_strlcat instead"))); ``` REASON: Enabling _FORTIFY_SOURCE will cause the compiler to use `strcpy()` with check, it results in a deprecated attribute declaration after including <features.h>. SOLUTION: move the deprecated attribute declaration from config.h to fmacro.h before "#include <features.h>". 3) networking.c COMPILER: GCC-12 WARNING: ``` networking.c: In function ‘addReplyDouble.part.0’: networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 876 | dbuf[start] = '$'; | ^ networking.c:868:14: note: at offset -5 into destination object ‘dbuf’ of size 5152 868 | char dbuf[MAX_LONG_DOUBLE_CHARS+32]; | ^ networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 876 | dbuf[start] = '$'; | ^ networking.c:868:14: note: at offset -6 into destination object ‘dbuf’ of size 5152 868 | char dbuf[MAX_LONG_DOUBLE_CHARS+32]; ``` REASON: GCC-12 predicts that digits10() may return 9 or 10 through `return 9 + (v >= 1000000000UL)`. SOLUTION: add an assert to let the compiler know the possible length; 4) redis-cli.c & redis-benchmark.c COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` redis-benchmark.c:1621:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL redis-cli.c:3015:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL ``` REASON: when _FORTIFY_SOURCE is enabled, the compiler will use the print() with check, which is a macro. this may result in the use of directives within the macro, which is undefined behavior. SOLUTION: move the directives-related code out of `print()`. 5) server.c COMPILER: gcc-13 with FORTIFY_SOURCE WARNING: ``` In function 'lookupCommandLogic', inlined from 'lookupCommandBySdsLogic' at server.c:3139:32: server.c:3102:66: error: '*(robj **)argv' may be used uninitialized [-Werror=maybe-uninitialized] 3102 | struct redisCommand *base_cmd = dictFetchValue(commands, argv[0]->ptr); | ~~~~^~~ ``` REASON: The compiler thinks that the `argc` returned by `sdssplitlen()` could be 0, resulting in an empty array of size 0 being passed to lookupCommandLogic. this should be a false positive, `argc` can't be 0 when strings are not NULL. SOLUTION: add an assert to let the compiler know that `argc` is positive. 6) sha1.c COMPILER: gcc-12 WARNING: ``` In function ‘SHA1Update’, inlined from ‘SHA1Final’ at sha1.c:195:5: sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread] 152 | SHA1Transform(context->state, &data[i]); | ^ sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’ sha1.c: In function ‘SHA1Final’: sha1.c:56:6: note: in a call to function ‘SHA1Transform’ 56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64]) | ^ In function ‘SHA1Update’, inlined from ‘SHA1Final’ at sha1.c:198:9: sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread] 152 | SHA1Transform(context->state, &data[i]); | ^ sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’ sha1.c: In function ‘SHA1Final’: sha1.c:56:6: note: in a call to function ‘SHA1Transform’ 56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64]) ``` REASON: due to the bug[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80922], when enable LTO, gcc-12 will not see `diagnostic ignored "-Wstringop-overread"`, resulting in a warning. SOLUTION: temporarily set SHA1Update to noinline to avoid compiler warnings due to LTO being enabled until the above gcc bug is fixed. 7) zmalloc.h COMPILER: GCC-12 WARNING: ``` In function ‘memset’, inlined from ‘moduleCreateContext’ at module.c:877:5, inlined from ‘RM_GetDetachedThreadSafeContext’ at module.c:8410:5: /usr/include/x86_64-linux-gnu/bits/string_fortified.h:59:10: warning: ‘__builtin_memset’ writing 104 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=] 59 | return __builtin___memset_chk (__dest, __ch, __len, ``` REASON: due to the GCC-12 bug [https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96503], GCC-12 cannot see alloc_size, which causes GCC to think that the actual size of memory is 0 when checking with __glibc_objsize0(). SOLUTION: temporarily set malloc-related interfaces to `noinline` to avoid compiler warnings due to LTO being enabled until the above gcc bug is fixed. ## Other changes 1) Fixed `ps -p [pid]` doesn't output `<defunct>` when using procps 4.x causing `replication child dies when parent is killed - diskless` test to fail. 2) Add a new fortify CI with GCC-13 and ubuntu-lunar docker image.
-
- 19 Mar, 2023 1 commit
-
-
Viktor Söderqvist authored
The message "Reading messages... (press Ctrl-C to quit)" is replaced by "Reading messages... (press Ctrl-C to quit or any key to type command)". This allows users to subscribe to more channels, to try out UNSUBSCRIBE and to combine pubsub with other features such as push messages from client tracking. The "Reading messages" info message is displayed in the bottom of the output in a distinct style and moves downward as more messages appear. When any key is pressed, the info message is replaced by the prompt with for entering commands. After entering a command and the reply is displayed, the "Reading messages" info messages appears again. This is added to the repl loop in redis-cli and in the corresponding place for non-interactive mode. An indication "(subscribed mode)" is included in the prompt when entering commands in subscribed mode. Also: * Fixes a problem that UNSUBSCRIBE hanged when used with RESP3 and push callback, without first entering subscribe mode. It hanged because UNSUBSCRIBE gets one or more push replies but no in-band reply. * Exit subscribed mode after RESET.
-
- 02 Feb, 2023 1 commit
-
-
Florian Weimer authored
Previously, jemalloc was explicitly configured to build in `gnu99` mode. As a result, `<stdatomic.h>` was presumed to be unavailable and never used. This commit removes explicit build flags configuration and lets `autoconf` determine the supported build flags. In addition, we also no longer build C++ jemalloc code. Co-authored-by:
Yossi Gottlieb <yossigo@gmail.com>
-
- 15 Oct, 2022 1 commit
-
-
filipe oliveira authored
optimizing d2string() and addReplyDouble() with grisu2: double to string conversion based on Florian Loitsch's Grisu-algorithm (#10587) All commands / use cases that heavily rely on double to a string representation conversion, (e.g. meaning take a double-precision floating-point number like 1.5 and return a string like "1.5" ), could benefit from a performance boost by swapping snprintf(buf,len,"%.17g",value) by the equivalent [fpconv_dtoa](https://github.com/night-shift/fpconv) or any other algorithm that ensures 100% coverage of conversion. This is a well-studied topic and Projects like MongoDB. RedPanda, PyTorch leverage libraries ( fmtlib ) that use the optimized double to string conversion underneath. The positive impact can be substantial. This PR uses the grisu2 approach ( grisu explained on https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf section 5 ). test suite changes: Despite being compatible, in some cases it produces a different result from printf, and some tests had to be adjusted. one case is that `%.17g` (which means %e or %f which ever is shorter), chose to use `5000000000` instead of 5e+9, which sounds like a bug? In other cases, we changed TCL to compare numbers instead of strings to ignore minor rounding issues (`expr 0.8 == 0.79999999999999999`)
-
- 23 Aug, 2022 1 commit
-
-
Oran Agra authored
* Support BUILD_TLS=module to be loaded as a module via config file or command line. e.g. redis-server --loadmodule redis-tls.so * Updates to redismodule.h to allow it to be used side by side with server.h by defining REDISMODULE_CORE_MODULE * Changes to server.h, redismodule.h and module.c to avoid repeated type declarations (gcc 4.8 doesn't like these) * Add a mechanism for non-ABI neutral modules (ones who include server.h) to refuse loading if they detect not being built together with redis (release.c) * Fix wrong signature of RedisModuleDefragFunc, this could break compilation of a module, but not the ABI * Move initialization of listeners in server.c to be after loading the modules * Config TLS after initialization of listeners * Init cluster after initialization of listeners * Add TLS module to CI * Fix a test suite race conditions: Now that the listeners are initialized later, it's not sufficient to wait for the PID message in the log, we need to wait for the "Server Initialized" message. * Fix issues with moduleconfigs test as a result from start_server waiting for "Server Initialized" * Fix issues with modules/infra test as a result of an additional module present Notes about Sentinel: Sentinel can't really rely on the tls module, since it uses hiredis to initiate connections and depends on OpenSSL (won't be able to use any other connection modules for that), so it was decided that when TLS is built as a module, sentinel does not support TLS at all. This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly. Example code of config in redis-tls.so(may be use in the future): RedisModuleString *tls_cfg = NULL; void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) { UNUSED(for_crash_report); RedisModule_InfoAddSection(ctx, ""); RedisModule_InfoAddFieldLongLong(ctx, "var", 42); } int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 2) return RedisModule_WrongArity(ctx); return RedisModule_ReplyWithString(ctx, argv[1]); } RedisModuleString *getStringConfigCommand(const char *name, void *privdata) { REDISMODULE_NOT_USED(name); REDISMODULE_NOT_USED(privdata); return tls_cfg; } int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) { REDISMODULE_NOT_USED(name); REDISMODULE_NOT_USED(err); REDISMODULE_NOT_USED(privdata); if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg); RedisModule_RetainString(NULL, new); tls_cfg = new; return REDISMODULE_OK; } int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc) { .... if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) { if (tls_cfg) { RedisModule_FreeString(ctx, tls_cfg); tls_cfg = NULL; } return REDISMODULE_ERR; } ... } Co-authored-by:
zhenwei pi <pizhenwei@bytedance.com> Signed-off-by:
zhenwei pi <pizhenwei@bytedance.com>
-
- 14 Aug, 2022 1 commit
-
-
Ozan Tezcan authored
Fix Lua compile warning on GCC 12.1 GCC 12.1 prints a warning on compile: ``` ldump.c: In function ‘DumpString’: ldump.c:63:26: warning: the comparison will always evaluate as ‘false’ for the pointer operand in ‘s + 24’ must not be NULL [-Waddress] 63 | if (s==NULL || getstr(s)==NULL) ``` It seems correct, `getstr(s)` can't be `NULL`. Also, I see Lua v5.2 does not have that check: https://github.com/lua/lua/blob/v5-2/ldump.c#L63
-
- 15 Jun, 2022 1 commit
-
-
Chris Lamb authored
-
- 01 Jun, 2022 1 commit
-
-
Ozan Tezcan authored
Apparently, GCC 11.2.0 has a new fancy warning for misleading indentations. It prints a warning when BRET(b) is on the same line as the loop.
-
- 22 May, 2022 1 commit
-
-
yoav-steinberg authored
The code is based on upstream https://github.com/HdrHistogram/HdrHistogram_c master branch latest commit (e4448cf6d1cd08fff519812d3b1e58bd5a94ac42). The reason to pull this in now is that their final version of our optimization is even faster. See: https://github.com/HdrHistogram/HdrHistogram_c/pull/107.
-
- 26 Apr, 2022 2 commits
-
-
meir authored
The white list is done by setting a metatable on the global table before initializing any library. The metatable set the `__newindex` field to a function that check the white list before adding the field to the table. Fields which is not on the white list are simply ignored. After initialization phase is done we protect the global table and each table that might be reachable from the global table. For each table we also protect the table metatable if exists.
-
meir authored
The new feature can be turned off and on using the new `lua_enablereadonlytable` Lua API.
-
- 20 Apr, 2022 1 commit
-
-
yoav-steinberg authored
`hdr_value_at_percentile()` is part of the Hdr_Histogram library used when generating `latencystats` report. There's a pending optimization for this function which greatly affects the performance of `info latencystats`. https://github.com/HdrHistogram/HdrHistogram_c/pull/107 This PR: 1. Upgrades the sources in _deps/hdr_histogram_ to the latest Hdr_Histogram version 0.11.5 2. Applies the referenced optimization. 3. Adds minor documentation about the hdr_histogram dependency which was missing under _deps/README.md_. benchmark on my machine: running: `redis-benchmark -n 100000 info latencystats` on a clean build with no data. | benchmark | RPS | | ---- | ---- | | before upgrade to v0.11.05 | 7,681 | | before optimization | 12,474 | | after optimization | 52,606 | Co-authored-by:
filipe oliveira <filipecosta.90@gmail.com>
-
- 14 Feb, 2022 1 commit
-
-
michael-grunder authored
Reapply this commit on top of hiredis as a local change. Previosuly it was pulled from a private hiredis branch, which resulted with it going away on subtree pull.
-
- 14 Jan, 2022 1 commit
-
-
Peter Hickey authored
-
- 05 Jan, 2022 1 commit
-
-
filipe oliveira authored
Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462) # Short description The Redis extended latency stats track per command latencies and enables: - exporting the per-command percentile distribution via the `INFO LATENCYSTATS` command. **( percentile distribution is not mergeable between cluster nodes ).** - exporting the per-command cumulative latency distributions via the `LATENCY HISTOGRAM` command. Using the cumulative distribution of latencies we can merge several stats from different cluster nodes to calculate aggregate metrics . By default, the extended latency monitoring is enabled since the overhead of keeping track of the command latency is very small. If you don't want to track extended latency metrics, you can easily disable it at runtime using the command: - `CONFIG SET latency-tracking no` By default, the exported latency percentiles are the p50, p99, and p999. You can alter them at runtime using the command: - `CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"` ## Some details: - The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command was called for the first time. - With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable vs this branch. - We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf ) ## `INFO LATENCYSTATS` exposition format - Format: `latency_percentiles_usec_<CMDNAME>:p0=XX,p50....` ## `LATENCY HISTOGRAM [command ...]` exposition format Return a cumulative distribution of latencies in the format of a histogram for the specified command names. The histogram is composed of a map of time buckets: - Each representing a latency range, between 1 nanosecond and roughly 1 second. - Each bucket covers twice the previous bucket's range. - Empty buckets are not printed. - Everything above 1 sec is considered +Inf. - At max there will be log2(1000000000)=30 buckets We reply a map for each command in the format: `<command name> : { `calls`: <total command calls> , `histogram` : { <bucket 1> : latency , < bucket 2> : latency, ... } }` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 29 Nov, 2021 1 commit
-
-
yoav-steinberg authored
caused a build warning in linenoise since glibc 2.20
-
- 28 Nov, 2021 1 commit
-
-
Meir Shpilraien (Spielrein) authored
msgpack lib missed using lua_checkstack and so on rare cases overflow the stack by at most 2 elements. This is a violation of the Lua C API. Notice that Lua allocates additional 5 more elements on top of lua->stack_last so Redis does not access an invalid memory. But it is an API violation and we should avoid it. This PR also added a new Lua compilation option. The new option can be enable using environment variable called LUA_DEBUG. If set to `yes` (by default `no`), Lua will be compiled without optimizations and with debug symbols (`-O0 -g`). In addition, in this new mode, Lua will be compiled with the `-DLUA_USE_APICHECK` flag that enables extended Lua C API validations. In addition, set LUA_DEBUG=yes on daily valgrind flow so we will be able to catch Lua C API violations in the future.
-
- 21 Nov, 2021 1 commit
-
-
Oran Agra authored
Background: Following the upgrade to jemalloc 5.2, there was a test that used to be flaky and started failing consistently (on 32bit), so we disabled it (see #9645). This is a test that i introduced in #7289 when i attempted to solve a rare stagnation problem, and it later turned out i failed to solve it, ans what's more i added a test that caused it to be not so rare, and as i mentioned, now in jemalloc 5.2 it became consistent on 32bit. Stagnation can happen when all the slabs of the bin are equally utilized, so the decision to move an allocation from a relatively empty slab to a relatively full one, will never happen, and in that test all the slabs are at 50% utilization, so the defragger could just keep scanning the keyspace and not move anything. What this PR changes: * First, finally in jemalloc 5.2 we have the count of non-full slabs, so when we compare the utilization of the current slab, we can compare it to the average utilization of the non-full slabs in our bin, instead of the total average of our bin. this takes the full slabs out of the game, since they're not candidates for migration (neither source nor target). * Secondly, We add some 12% (100/8) to the decision to defrag an allocation, this is the part that aims to avoid stagnation, and it's especially important since the above mentioned change can get us closer to stagnation. * Thirdly, since jemalloc 5.2 adds sharded bins, we take into account all shards (something that's missing from the original PR that merged it), this isn't expected to make any difference since anyway there should be just one shard. How this was benchmarked. What i did was run the memefficiency test unit with `--verbose` and compare the defragger hits and misses the tests reported. At first, when i took into consideration only the non-full slabs, it got a lot worse (i got into stagnation, or just got a lot of misses and a lot of hits), but when i added the 10% i got back to results that were slightly better than the ones of the jemalloc 5.1 branch. i.e. full defragmentation was achieved with fewer hits (relocations), and fewer misses (keyspace scans).
-
- 27 Oct, 2021 1 commit
-
-
yoav-steinberg authored
We now use git subtree for deps/jemalloc, updating jemalloc is detailed in deps/README.md
-
- 24 Oct, 2021 1 commit
-
-
Yossi Gottlieb authored
-
- 12 Oct, 2021 5 commits
-
-
Yossi Gottlieb authored
Cherry pick a more complete fix to 0215324a that also doesn't leak memory from latest hiredis.
-
Yoav Steinberg authored
-
Oran Agra authored
-
YoongHM authored
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default. All it does is avoid updating unchanged objects (shouldn't have any impact on our build) - Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead code in redis anyway). - Remove unused variable `c` in `f_parser()` - Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Oran Agra authored
There's a rare case which leads to stagnation in the defragger, causing it to keep scanning the keyspace and do nothing (not moving any allocation), this happens when all the allocator slabs of a certain bin have the same % utilization, but the slab from which new allocations are made have a lower utilization. this commit fixes it by removing the current slab from the overall average utilization of the bin, and also eliminate any precision loss in the utilization calculation and move the decision about the defrag to reside inside jemalloc. and also add a test that consistently reproduce this issue.
-
- 10 Oct, 2021 4 commits
-
-
Oran Agra authored
* overflow in jemalloc fragmentation hint to the defragger
-
Oran Agra authored
-
Yoav Steinberg authored
./autogen.sh --with-version=5.2.1-0-g0