1. 02 Oct, 2019 1 commit
    • Oran Agra's avatar
      RED-31295 - redis: avoid race between dlopen and thread creation · 2e19b941
      Oran Agra authored
      It seeems that since I added the creation of the jemalloc thread redis
      sometimes fails to start with the following error:
      
      Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!
      
      This seems to be due to a race bug in ld.so, in which TLS creation on the
      thread, collide with dlopen.
      
      Move the creation of BIO and jemalloc threads to after modules are loaded.
      
      plus small bugfix when trying to disable the jemalloc thread at runtime
      2e19b941
  2. 02 Jun, 2019 1 commit
    • Oran Agra's avatar
      make redis purge jemalloc after flush, and enable background purging thread · 09f99c2a
      Oran Agra authored
      jemalloc 5 doesn't immediately release memory back to the OS, instead there's a decaying
      mechanism, which doesn't work when there's no traffic (no allocations).
      this is most evident if there's no traffic after flushdb, the RSS will remain high.
      
      1) enable jemalloc background purging
      2) explicitly purge in flushdb
      09f99c2a
  3. 21 Mar, 2019 1 commit
  4. 30 Sep, 2018 1 commit
    • Bruce Merry's avatar
      Fix incorrect memory usage accounting in zrealloc · 8fd1031b
      Bruce Merry authored
      When HAVE_MALLOC_SIZE is false, each call to zrealloc causes used_memory
      to increase by PREFIX_SIZE more than it should, due to mis-matched
      accounting between the original zmalloc (which includes PREFIX size in
      its increment) and zrealloc (which misses it from its decrement).
      
      I've also supplied a command-line test to easily demonstrate the
      problem. It's not wired into the test framework, because I don't know
      TCL so I'm not sure how to automate it.
      8fd1031b
  5. 22 Jul, 2018 1 commit
  6. 16 Jul, 2018 1 commit
    • Oran Agra's avatar
      slave buffers were wasteful and incorrectly counted causing eviction · bf680b6f
      Oran Agra authored
      A) slave buffers didn't count internal fragmentation and sds unused space,
         this caused them to induce eviction although we didn't mean for it.
      
      B) slave buffers were consuming about twice the memory of what they actually needed.
      - this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
        but networking.c not storing more than 16k (partially fixed recently in 237a38737).
      - besides it wasn't able to store half of the new string into one buffer and the
        other half into the next (so the above mentioned fix helped mainly for small items).
      - lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
        consumed but not used.
      
      C) inefficient performance due to starting from a small string and reallocing many times.
      
      what i changed:
      - creating dedicated buffers for reply list, counting their size with zmalloc_size
      - when creating a new reply node from, preallocate it to at least 16k.
      - when appending a new reply to the buffer, first fill all the unused space of the
        previous node before starting a new one.
      
      other changes:
      - expose mem_not_counted_for_evict info field for the benefit of the test suite
      - add a test to make sure slave buffers are counted correctly and that they don't cause eviction
      bf680b6f
  7. 03 Jul, 2018 1 commit
  8. 14 Jun, 2018 1 commit
  9. 30 May, 2018 1 commit
  10. 17 May, 2018 1 commit
    • Oran Agra's avatar
      Active defrag fixes for 32bit builds · ad133e10
      Oran Agra authored
      problems fixed:
      * failing to read fragmentation information from jemalloc
      * overflow in jemalloc fragmentation hint to the defragger
      * test suite not triggering eviction after population
      ad133e10
  11. 06 May, 2018 1 commit
  12. 12 Mar, 2018 1 commit
    • Oran Agra's avatar
      Adding real allocator fragmentation to INFO and MEMORY command + active defrag test · 806736cd
      Oran Agra authored
      other fixes / improvements:
      - LUA script memory isn't taken from zmalloc (taken from libc malloc)
        so it can cause high fragmentation ratio to be displayed (which is false)
      - there was a problem with "fragmentation" info being calculated from
        RSS and used_memory sampled at different times (now sampling them together)
      
      other details:
      - adding a few more allocator info fields to INFO and MEMORY commands
      - improve defrag test to measure defrag latency of big keys
      - increasing the accuracy of the defrag test (by looking at real grag info)
        this way we can use an even lower threshold and still avoid false positives
      - keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
      - add these the MEMORY DOCTOR command
      - deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
      - reduce sampling rate of the rss and allocator info
      806736cd
  13. 21 Feb, 2018 1 commit
    • Oran Agra's avatar
      Fix zrealloc to behave similarly to je_realloc when size is 0 · 5def6500
      Oran Agra authored
      According to C11, the behavior of realloc with size 0 is now deprecated.
      it can either behave as free(ptr) and return NULL, or return a valid pointer.
      but in zmalloc it can lead to zmalloc_oom_handler and panic.
      and that can affect modules that use it.
      
      It looks like both glibc allocator and jemalloc behave like so:
        realloc(malloc(32),0) returns NULL
        realloc(NULL,0) returns a valid pointer
      
      This commit changes zmalloc to behave the same
      5def6500
  14. 09 May, 2017 1 commit
  15. 04 May, 2017 1 commit
  16. 11 Apr, 2017 2 commits
  17. 10 Jan, 2017 1 commit
  18. 30 Dec, 2016 1 commit
  19. 19 Sep, 2016 2 commits
  20. 13 Jan, 2016 1 commit
  21. 01 Oct, 2015 1 commit
  22. 17 Dec, 2014 2 commits
  23. 14 Nov, 2014 1 commit
  24. 12 Nov, 2014 1 commit
  25. 26 Jun, 2014 2 commits
  26. 24 Mar, 2014 1 commit
    • antirez's avatar
      Sample and cache RSS in serverCron(). · 93253c27
      antirez authored
      Obtaining the RSS (Resident Set Size) info is slow in Linux and OSX.
      This slowed down the generation of the INFO 'memory' section.
      
      Since the RSS does not require to be a real-time measurement, we
      now sample it with server.hz frequency (10 times per second by default)
      and use this value both to show the INFO rss field and to compute the
      fragmentation ratio.
      
      Practically this does not make any difference for memory profiling of
      Redis but speeds up the INFO call significantly.
      93253c27
  27. 05 Dec, 2013 1 commit
  28. 19 Jan, 2013 1 commit
  29. 21 Nov, 2012 1 commit
  30. 19 Nov, 2012 1 commit
  31. 14 Nov, 2012 1 commit
  32. 24 Aug, 2012 2 commits
    • antirez's avatar
    • antirez's avatar
      Better Out of Memory handling. · 6fdc6354
      antirez authored
      The previous implementation of zmalloc.c was not able to handle out of
      memory in an application-specific way. It just logged an error on
      standard error, and aborted.
      
      The result was that in the case of an actual out of memory in Redis
      where malloc returned NULL (In Linux this actually happens under
      specific overcommit policy settings and/or with no or little swap
      configured) the error was not properly logged in the Redis log.
      
      This commit fixes this problem, fixing issue #509.
      Now the out of memory is properly reported in the Redis log and a stack
      trace is generated.
      
      The approach used is to provide a configurable out of memory handler
      to zmalloc (otherwise the default one logging the event on the
      standard output is used).
      6fdc6354
  33. 27 Mar, 2012 2 commits
  34. 26 Feb, 2012 1 commit