- 26 Oct, 2020 1 commit
-
-
Yossi Gottlieb authored
When using a system with no malloc_usable_size(), zmalloc_size() assumed that the heap allocator always returns blocks that are long-padded. This may not always be the case, and will result with zmalloc_size() returning a size that is bigger than allocated. At least in one case this leads to out of bound write, process crash and a potential security vulnerability. Effectively this does not affect the vast majority of users, who use jemalloc or glibc. This problem along with a (different) fix was reported by Drew DeVault.
-
- 02 Oct, 2020 1 commit
-
-
Oran Agra authored
This commit has two aspects: 1) improve memory reporting for all the places that use sdsAllocSize to compute memory used by a string, in this case it'll include the internal fragmentation. 2) reduce the need for realloc calls by making the sds implicitly take over the internal fragmentation of the block it allocated.
-
- 29 Sep, 2020 1 commit
-
-
David CARLIER authored
-
- 17 Sep, 2020 1 commit
-
-
Wang Yuan authored
Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11 _Atomic to establish inter-thread synchronization without mutex. But the compiler that must supports C11 _Atomic can compile redis code, that brings a lot of inconvenience since some common platforms can't support by default such as CentOS7, so we want to implement redis atomic type to make it more portable. We have implemented our atomic variable for redis that only has 'relaxed' operations in src/atomicvar.h, so we implement some operations with 'sequentially-consistent', just like the default behavior of C11 _Atomic that can establish inter-thread synchronization. And we replace all uses of C11 _Atomic with redis atomic variable. Our implementation of redis atomic variable uses C11 _Atomic, __atomic or __sync macros if available, it supports most common platforms, and we will detect automatically which feature we use. In Makefile we use a dummy file to detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis code theoretically if your gcc version is not less than 4.1.2(starts to support __sync_xxx operations). Otherwise, we remove use mutex fallback to implement redis atomic variable for performance and test. You will get compiling errors if your compiler doesn't support all features of above. For cover redis atomic variable tests, we add other CI jobs that build redis on CentOS6 and CentOS7 and workflow daily jobs that run the tests on them. For them, we just install gcc by default in order to cover different compiler versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7. We restore the feature that we can test redis with Helgrind to find data race errors. But you need install Valgrind in the default path configuration firstly before running your tests, since we use macros in helgrind.h to tell Helgrind inter-thread happens-before relationship explicitly for avoiding false positives. Please open an issue on github if you find data race errors relate to this commit. Unrelated: - Fix redefinition of typedef 'RedisModuleUserChangedFunc' For some old version compilers, they will report errors or warnings, if we re-define function type.
-
- 31 Jul, 2020 1 commit
-
- 02 Dec, 2019 1 commit
-
-
antirez authored
-
- 02 Oct, 2019 1 commit
-
-
Oran Agra authored
It seeems that since I added the creation of the jemalloc thread redis sometimes fails to start with the following error: Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed! This seems to be due to a race bug in ld.so, in which TLS creation on the thread, collide with dlopen. Move the creation of BIO and jemalloc threads to after modules are loaded. plus small bugfix when trying to disable the jemalloc thread at runtime
-
- 20 Sep, 2019 1 commit
-
-
David Carlier authored
-
- 15 Sep, 2019 1 commit
-
-
David Carlier authored
-
- 28 Jul, 2019 1 commit
-
-
David Carlier authored
-
- 02 Jun, 2019 1 commit
-
-
Oran Agra authored
jemalloc 5 doesn't immediately release memory back to the OS, instead there's a decaying mechanism, which doesn't work when there's no traffic (no allocations). this is most evident if there's no traffic after flushdb, the RSS will remain high. 1) enable jemalloc background purging 2) explicitly purge in flushdb
-
- 21 Mar, 2019 1 commit
-
-
antirez authored
-
- 30 Sep, 2018 1 commit
-
-
Bruce Merry authored
When HAVE_MALLOC_SIZE is false, each call to zrealloc causes used_memory to increase by PREFIX_SIZE more than it should, due to mis-matched accounting between the original zmalloc (which includes PREFIX size in its increment) and zrealloc (which misses it from its decrement). I've also supplied a command-line test to easily demonstrate the problem. It's not wired into the test framework, because I don't know TCL so I'm not sure how to automate it.
-
- 22 Jul, 2018 1 commit
-
-
Oran Agra authored
-
- 16 Jul, 2018 1 commit
-
-
Oran Agra authored
A) slave buffers didn't count internal fragmentation and sds unused space, this caused them to induce eviction although we didn't mean for it. B) slave buffers were consuming about twice the memory of what they actually needed. - this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time but networking.c not storing more than 16k (partially fixed recently in 237a38737). - besides it wasn't able to store half of the new string into one buffer and the other half into the next (so the above mentioned fix helped mainly for small items). - lastly, the sds buffers had up to 30% internal fragmentation that was wasted, consumed but not used. C) inefficient performance due to starting from a small string and reallocing many times. what i changed: - creating dedicated buffers for reply list, counting their size with zmalloc_size - when creating a new reply node from, preallocate it to at least 16k. - when appending a new reply to the buffer, first fill all the unused space of the previous node before starting a new one. other changes: - expose mem_not_counted_for_evict info field for the benefit of the test suite - add a test to make sure slave buffers are counted correctly and that they don't cause eviction
-
- 03 Jul, 2018 1 commit
-
-
Jack Drogon authored
-
- 14 Jun, 2018 1 commit
-
-
Fuxin Hao authored
-
- 30 May, 2018 1 commit
-
-
Remi Collet authored
-
- 17 May, 2018 1 commit
-
-
Oran Agra authored
problems fixed: * failing to read fragmentation information from jemalloc * overflow in jemalloc fragmentation hint to the defragger * test suite not triggering eviction after population
-
- 06 May, 2018 1 commit
-
-
Krzysztof Filipek authored
-
- 12 Mar, 2018 1 commit
-
-
Oran Agra authored
other fixes / improvements: - LUA script memory isn't taken from zmalloc (taken from libc malloc) so it can cause high fragmentation ratio to be displayed (which is false) - there was a problem with "fragmentation" info being calculated from RSS and used_memory sampled at different times (now sampling them together) other details: - adding a few more allocator info fields to INFO and MEMORY commands - improve defrag test to measure defrag latency of big keys - increasing the accuracy of the defrag test (by looking at real grag info) this way we can use an even lower threshold and still avoid false positives - keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things - add these the MEMORY DOCTOR command - deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used") - reduce sampling rate of the rss and allocator info
-
- 21 Feb, 2018 1 commit
-
-
Oran Agra authored
According to C11, the behavior of realloc with size 0 is now deprecated. it can either behave as free(ptr) and return NULL, or return a valid pointer. but in zmalloc it can lead to zmalloc_oom_handler and panic. and that can affect modules that use it. It looks like both glibc allocator and jemalloc behave like so: realloc(malloc(32),0) returns NULL realloc(NULL,0) returns a valid pointer This commit changes zmalloc to behave the same
-
- 09 May, 2017 1 commit
-
-
antirez authored
-
- 04 May, 2017 1 commit
-
-
antirez authored
-
- 11 Apr, 2017 2 commits
- 10 Jan, 2017 1 commit
-
-
antirez authored
This commit also includes minor aesthetic changes like removal of trailing spaces.
-
- 30 Dec, 2016 1 commit
-
-
oranagra authored
-
- 19 Sep, 2016 2 commits
- 13 Jan, 2016 1 commit
-
-
root authored
-
- 01 Oct, 2015 1 commit
-
-
antirez authored
-
- 17 Dec, 2014 2 commits
- 14 Nov, 2014 1 commit
-
-
antirez authored
-
- 12 Nov, 2014 1 commit
-
-
antirez authored
-
- 26 Jun, 2014 2 commits
-
-
Matt Stancliff authored
From mailing list post https://groups.google.com/forum/#!topic/redis-db/QLjiQe4D7LA In zmalloc.c the following primitives are currently used to synchronize access to single global variable: __sync_add_and_fetch __sync_sub_and_fetch In some architectures such as powerpc these primitives are overhead intensive. More efficient C11 __atomic builtins are available with newer GCC versions, see http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/_005f_005fatomic-Builtins.html#_005f_005fatomic-Builtins By substituting the following __atomic… builtins: __atomic_add_fetch __atomic_sub_fetch the performance improvement on certain architectures such as powerpc can be significant, around 10% to 15%, over the implementation using __sync builtins while there is only slight uptick on Intel architectures because it was already enforcing Intel Strongly ordered memory semantics. The selection of __atomic built-ins can be predicated on the definition of ATOMIC_RELAXED which Is available on in gcc 4.8.2 and later versions.
-
Matt Stancliff authored
-
- 24 Mar, 2014 1 commit
-
-
antirez authored
Obtaining the RSS (Resident Set Size) info is slow in Linux and OSX. This slowed down the generation of the INFO 'memory' section. Since the RSS does not require to be a real-time measurement, we now sample it with server.hz frequency (10 times per second by default) and use this value both to show the INFO rss field and to compute the fragmentation ratio. Practically this does not make any difference for memory profiling of Redis but speeds up the INFO call significantly.
-
- 05 Dec, 2013 1 commit
-
-
antirez authored
-