- 01 May, 2023 1 commit
-
-
Oran Agra authored
* Regenerate configure script sccording to deps/README * update iget_defrag_hint by following changes to arena_dalloc_no_tcache
-
- 18 Apr, 2023 1 commit
-
-
sundb authored
This PR is to fix the compilation warnings and errors generated by the latest complier toolchain, and to add a new runner of the latest toolchain for daily CI. ## Fix various compilation warnings and errors 1) jemalloc.c COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` src/jemalloc.c:1028:7: warning: suspicious concatenation of string literals in an array initialization; did you mean to separate the elements with a comma? [-Wstring-concatenation] "/etc/malloc.conf", ^ src/jemalloc.c:1027:3: note: place parentheses around the string literal to silence warning "\"name\" of the file referenced by the symbolic link named " ^ ``` REASON: the compiler to alert developers to potential issues with string concatenation that may miss a comma, just like #9534 which misses a comma. SOLUTION: use `()` to tell the compiler that these two line strings are continuous. 2) config.h COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` In file included from quicklist.c:36: ./config.h:319:76: warning: attribute declaration must precede definition [-Wignored-attributes] char *strcat(char *restrict dest, const char *restrict src) __attribute__((deprecated("please avoid use of unsafe C functions. prefer use of redis_strlcat instead"))); ``` REASON: Enabling _FORTIFY_SOURCE will cause the compiler to use `strcpy()` with check, it results in a deprecated attribute declaration after including <features.h>. SOLUTION: move the deprecated attribute declaration from config.h to fmacro.h before "#include <features.h>". 3) networking.c COMPILER: GCC-12 WARNING: ``` networking.c: In function ‘addReplyDouble.part.0’: networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 876 | dbuf[start] = '$'; | ^ networking.c:868:14: note: at offset -5 into destination object ‘dbuf’ of size 5152 868 | char dbuf[MAX_LONG_DOUBLE_CHARS+32]; | ^ networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 876 | dbuf[start] = '$'; | ^ networking.c:868:14: note: at offset -6 into destination object ‘dbuf’ of size 5152 868 | char dbuf[MAX_LONG_DOUBLE_CHARS+32]; ``` REASON: GCC-12 predicts that digits10() may return 9 or 10 through `return 9 + (v >= 1000000000UL)`. SOLUTION: add an assert to let the compiler know the possible length; 4) redis-cli.c & redis-benchmark.c COMPILER: clang-14 with FORTIFY_SOURCE WARNING: ``` redis-benchmark.c:1621:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL redis-cli.c:3015:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL ``` REASON: when _FORTIFY_SOURCE is enabled, the compiler will use the print() with check, which is a macro. this may result in the use of directives within the macro, which is undefined behavior. SOLUTION: move the directives-related code out of `print()`. 5) server.c COMPILER: gcc-13 with FORTIFY_SOURCE WARNING: ``` In function 'lookupCommandLogic', inlined from 'lookupCommandBySdsLogic' at server.c:3139:32: server.c:3102:66: error: '*(robj **)argv' may be used uninitialized [-Werror=maybe-uninitialized] 3102 | struct redisCommand *base_cmd = dictFetchValue(commands, argv[0]->ptr); | ~~~~^~~ ``` REASON: The compiler thinks that the `argc` returned by `sdssplitlen()` could be 0, resulting in an empty array of size 0 being passed to lookupCommandLogic. this should be a false positive, `argc` can't be 0 when strings are not NULL. SOLUTION: add an assert to let the compiler know that `argc` is positive. 6) sha1.c COMPILER: gcc-12 WARNING: ``` In function ‘SHA1Update’, inlined from ‘SHA1Final’ at sha1.c:195:5: sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread] 152 | SHA1Transform(context->state, &data[i]); | ^ sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’ sha1.c: In function ‘SHA1Final’: sha1.c:56:6: note: in a call to function ‘SHA1Transform’ 56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64]) | ^ In function ‘SHA1Update’, inlined from ‘SHA1Final’ at sha1.c:198:9: sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread] 152 | SHA1Transform(context->state, &data[i]); | ^ sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’ sha1.c: In function ‘SHA1Final’: sha1.c:56:6: note: in a call to function ‘SHA1Transform’ 56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64]) ``` REASON: due to the bug[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80922], when enable LTO, gcc-12 will not see `diagnostic ignored "-Wstringop-overread"`, resulting in a warning. SOLUTION: temporarily set SHA1Update to noinline to avoid compiler warnings due to LTO being enabled until the above gcc bug is fixed. 7) zmalloc.h COMPILER: GCC-12 WARNING: ``` In function ‘memset’, inlined from ‘moduleCreateContext’ at module.c:877:5, inlined from ‘RM_GetDetachedThreadSafeContext’ at module.c:8410:5: /usr/include/x86_64-linux-gnu/bits/string_fortified.h:59:10: warning: ‘__builtin_memset’ writing 104 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=] 59 | return __builtin___memset_chk (__dest, __ch, __len, ``` REASON: due to the GCC-12 bug [https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96503], GCC-12 cannot see alloc_size, which causes GCC to think that the actual size of memory is 0 when checking with __glibc_objsize0(). SOLUTION: temporarily set malloc-related interfaces to `noinline` to avoid compiler warnings due to LTO being enabled until the above gcc bug is fixed. ## Other changes 1) Fixed `ps -p [pid]` doesn't output `<defunct>` when using procps 4.x causing `replication child dies when parent is killed - diskless` test to fail. 2) Add a new fortify CI with GCC-13 and ubuntu-lunar docker image.
-
- 21 Nov, 2021 1 commit
-
-
Oran Agra authored
Background: Following the upgrade to jemalloc 5.2, there was a test that used to be flaky and started failing consistently (on 32bit), so we disabled it (see #9645). This is a test that i introduced in #7289 when i attempted to solve a rare stagnation problem, and it later turned out i failed to solve it, ans what's more i added a test that caused it to be not so rare, and as i mentioned, now in jemalloc 5.2 it became consistent on 32bit. Stagnation can happen when all the slabs of the bin are equally utilized, so the decision to move an allocation from a relatively empty slab to a relatively full one, will never happen, and in that test all the slabs are at 50% utilization, so the defragger could just keep scanning the keyspace and not move anything. What this PR changes: * First, finally in jemalloc 5.2 we have the count of non-full slabs, so when we compare the utilization of the current slab, we can compare it to the average utilization of the non-full slabs in our bin, instead of the total average of our bin. this takes the full slabs out of the game, since they're not candidates for migration (neither source nor target). * Secondly, We add some 12% (100/8) to the decision to defrag an allocation, this is the part that aims to avoid stagnation, and it's especially important since the above mentioned change can get us closer to stagnation. * Thirdly, since jemalloc 5.2 adds sharded bins, we take into account all shards (something that's missing from the original PR that merged it), this isn't expected to make any difference since anyway there should be just one shard. How this was benchmarked. What i did was run the memefficiency test unit with `--verbose` and compare the defragger hits and misses the tests reported. At first, when i took into consideration only the non-full slabs, it got a lot worse (i got into stagnation, or just got a lot of misses and a lot of hits), but when i added the 10% i got back to results that were slightly better than the ones of the jemalloc 5.1 branch. i.e. full defragmentation was achieved with fewer hits (relocations), and fewer misses (keyspace scans).
-
- 12 Oct, 2021 3 commits
-
-
Oran Agra authored
-
YoongHM authored
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default. All it does is avoid updating unchanged objects (shouldn't have any impact on our build) - Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead code in redis anyway). - Remove unused variable `c` in `f_parser()` - Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
Oran Agra authored
There's a rare case which leads to stagnation in the defragger, causing it to keep scanning the keyspace and do nothing (not moving any allocation), this happens when all the allocator slabs of a certain bin have the same % utilization, but the slab from which new allocations are made have a lower utilization. this commit fixes it by removing the current slab from the overall average utilization of the bin, and also eliminate any precision loss in the utilization calculation and move the decision about the defrag to reside inside jemalloc. and also add a test that consistently reproduce this issue.
-
- 10 Oct, 2021 5 commits
-
-
Oran Agra authored
* overflow in jemalloc fragmentation hint to the defragger
-
Oran Agra authored
-
Yoav Steinberg authored
./autogen.sh --with-version=5.2.1-0-g0
-
Yoav Steinberg authored
- 03 May, 2021 1 commit
-
-
Oran Agra authored
-
- 29 Sep, 2020 1 commit
-
-
YoongHM authored
- The argument `u` in for `ar` is ignored (and generates warnings since `D` became the default. All it does is avoid updating unchanged objects (shouldn't have any impact on our build) - Enable `LUA_USE_MKSTEMP` to force the use of `mkstemp()` instead of `tmpname()` (which is dead code in redis anyway). - Remove unused variable `c` in `f_parser()` - Removed misleadingly indented space in `luaL_loadfile()` and ``addfield()` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 21 Sep, 2020 1 commit
-
-
YoongHM authored
Change `val` to `unsigned char` before being tested. The fix is identical to the one that's been made in upstream jemalloc. warning is: src/malloc_io.c: In function ‘malloc_vsnprintf’: src/malloc_io.c:369:2: warning: case label value exceeds maximum value for type 369 | case '?' | 0x80: \ | ^~~~ src/malloc_io.c:581:5: note: in expansion of macro ‘GET_ARG_NUMERIC’ 581 | GET_ARG_NUMERIC(val, 'p'); | ^~~~~~~~~~~~~~~
-
- 20 May, 2020 1 commit
-
-
Oran Agra authored
There's a rare case which leads to stagnation in the defragger, causing it to keep scanning the keyspace and do nothing (not moving any allocation), this happens when all the allocator slabs of a certain bin have the same % utilization, but the slab from which new allocations are made have a lower utilization. this commit fixes it by removing the current slab from the overall average utilization of the bin, and also eliminate any precision loss in the utilization calculation and move the decision about the defrag to reside inside jemalloc. and also add a test that consistently reproduce this issue.
-
- 02 Jun, 2019 1 commit
-
-
Oran Agra authored
Background threads may run for a long time, especially when the # of dirty pages is high. Avoid blocking stats calls because of this (which may cause latency spikes). see https://github.com/jemalloc/jemalloc/issues/1502 cherry picked from commit 1a71533511027dbe3f9d989659efeec446915d6b
-
- 11 Jul, 2018 1 commit
-
-
Oran Agra authored
* overflow in jemalloc fragmentation hint to the defragger
-
- 27 Jun, 2018 1 commit
-
-
Oran Agra authored
-
- 24 May, 2018 2 commits
- 17 May, 2018 1 commit
-
-
Oran Agra authored
problems fixed: * failing to read fragmentation information from jemalloc * overflow in jemalloc fragmentation hint to the defragger * test suite not triggering eviction after population
-
- 22 Apr, 2017 1 commit
-
- 30 Jan, 2017 1 commit
-
-
antirez authored
The original jemalloc source tree was modified to: 1. Remove the configure error that prevents nested builds. 2. Insert the Redis private Jemalloc API in order to allow the Redis fragmentation function to work.
-
- 10 Jan, 2017 1 commit
-
-
antirez authored
This commit also includes minor aesthetic changes like removal of trailing spaces.
-
- 30 Dec, 2016 1 commit
-
-
oranagra authored
-
- 07 Oct, 2015 1 commit
-
-
antirez authored
Now way to make unmodified Jemalloc configure to work when the jemalloc source tree is inside a subdirectory of a different git repository. Problem signaled here: http://www.canonware.com/pipermail/jemalloc-discuss/2015-October/001174.html
-
- 06 Oct, 2015 1 commit
-
-
antirez authored
-
- 24 Jul, 2015 1 commit
-
-
antirez authored
This gives us a 24 bytes size class which is dict.c dictEntry size, thus improving the memory efficiency of Redis significantly. Moreover other non 16 bytes aligned tiny classes are added that further reduce the fragmentation of the allocator. Technically speaking LG_QUANTUM should be 4 on i386 / AMD64 because of SSE types and other 16 bytes types, however we don't use those, and our jemalloc only targets Redis. New versions of Jemalloc will have an explicit configure switch in order to specify the quantum value for a platform without requiring any change to the Jemalloc source code: we'll switch to this system when available. This change was originally proposed by Oran Agra (@oranagra) as a change to the Jemalloc script to generate the size classes define. We ended doing it differently by changing LG_QUANTUM since it is apparently the supported Jemalloc method to obtain a 24 bytes size class, moreover it also provides us other potentially useful size classes. Related to issue #2510.
-
- 20 Jun, 2014 1 commit
-
-
antirez authored
Not a single bug in about 3 months, and our previous version was too old (3.2.0).
-
- 18 Apr, 2013 1 commit
-
-
antirez authored
Redis gitignore was too aggressive since simply broken. Jemalloc gitignore was too agressive because it is conceived to just keep the files that allow to generate all the rest in development environments (so for instance the "configure" file is excluded).
-
- 28 Nov, 2012 1 commit
-
-
antirez authored
-
- 16 May, 2012 1 commit
-
-
antirez authored
Full changelog here: http://www.canonware.com/cgi-bin/gitweb.cgi?p=jemalloc.git;a=blob_plain;f=ChangeLog;hb=master Notable improvements from the point of view of Redis: 1) Bugfixing. 2) Support for Valgrind. 3) Support for OSX Lion, FreeBSD.
-
- 25 Nov, 2011 1 commit
-
-
antirez authored
-
- 23 Nov, 2011 1 commit
-
-
jbergstroem authored
-
- 20 Jun, 2011 2 commits
-
-
Pieter Noordhuis authored
-
antirez authored
-