1. 16 Nov, 2021 1 commit
  2. 04 Aug, 2021 1 commit
    • Wang Yuan's avatar
      Use madvise(MADV_DONTNEED) to release memory to reduce COW (#8974) · d4bca53c
      Wang Yuan authored
      
      
      ## Backgroud
      As we know, after `fork`, one process will copy pages when writing data to these
      pages(CoW), and another process still keep old pages, they totally cost more memory.
      For redis, we suffered that redis consumed much memory when the fork child is serializing
      key/values, even that maybe cause OOM.
      
      But actually we find, in redis fork child process, the child process don't need to keep some
      memory and parent process may write or update that, for example, child process will never
      access the key-value that is serialized but users may update it in parent process.
      So we think it may reduce COW if the child process release memory that it is not needed.
      
      ## Implementation
      For releasing key value in child process, we may think we call `decrRefCount` to free memory,
      but i find the fork child process still use much memory when we don't write any data to redis,
      and it costs much more time that slows down bgsave. Maybe because memory allocator doesn't
      really release memory to OS, and it may modify some inner data for this free operation, especially
      when we free small objects.
      
      Moreover, CoW is based on  pages, so it is a easy way that we only free the memory bulk that is
      not less than kernel page size. madvise(MADV_DONTNEED) can quickly release specified region
      pages to OS bypassing memory allocator, and allocator still consider that this memory still is used
      and don't change its inner data.
      
      There are some buffers we can release in the fork child process:
      - **Serialized key-values**
        the fork child process never access serialized key-values, so we try to free them.
        Because we only can release big bulk memory, and it is time consumed to iterate all
        items/members/fields/entries of complex data type. So we decide to iterate them and
        try to release them only when their average size of item/member/field/entry is more
        than page size of OS.
      - **Replication backlog**
        Because replication backlog is a cycle buffer, it will be changed quickly if redis has heavy
        write traffic, but in fork child process, we don't need to access that.
      - **Client buffers**
        If clients have requests during having the fork child process, clients' buffer also be changed
        frequently. The memory includes client query buffer, output buffer, and client struct used memory.
      
      To get child process peak private dirty memory, we need to count peak memory instead
      of last used memory, because the child process may continue to release memory (since
      COW used to only grow till now, the last was equivalent to the peak).
      Also we're adding a new `current_cow_peak` info variable (to complement the existing
      `current_cow_size`)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      d4bca53c
  3. 10 Mar, 2021 1 commit
    • sundb's avatar
      Add run all test support with define REDIS_TEST (#8570) · 95d6297d
      sundb authored
      1. Add `redis-server test all` support to run all tests.
      2. Add redis test to daily ci.
      3. Add `--accurate` option to run slow tests for more iterations (so that
         by default we run less cycles (shorter time, and less prints).
      4. Move dict benchmark to REDIS_TEST.
      5. fix some leaks in tests
      6. make quicklist tests run on a specific fill set of options rather than huge ranges
      7. move some prints in quicklist test outside their loops to reduce prints
      8. removing sds.h from dict.c since it is now used in both redis-server and
         redis-cli (uses hiredis sds)
      95d6297d
  4. 07 Mar, 2021 1 commit
  5. 25 Feb, 2021 1 commit
    • Yossi Gottlieb's avatar
      Cleanup usage of malloc_usable_size. (#8554) · 3ea4c43a
      Yossi Gottlieb authored
      * Add better control of malloc_usable_size() usage.
      * Use malloc_usable_size on alpine libc daily job.
      * Add no-malloc-usable-size daily jobs.
      * Fix zmalloc(0) when HAVE_MALLOC_SIZE is undefined.
      
      In order to align with the jemalloc behavior, this should never return
      NULL or OOM panic.
      3ea4c43a
  6. 24 Feb, 2021 1 commit
  7. 06 Dec, 2020 1 commit
  8. 02 Oct, 2020 1 commit
  9. 02 Dec, 2019 1 commit
  10. 15 Sep, 2019 1 commit
  11. 02 Jun, 2019 1 commit
    • Oran Agra's avatar
      make redis purge jemalloc after flush, and enable background purging thread · 09f99c2a
      Oran Agra authored
      jemalloc 5 doesn't immediately release memory back to the OS, instead there's a decaying
      mechanism, which doesn't work when there's no traffic (no allocations).
      this is most evident if there's no traffic after flushdb, the RSS will remain high.
      
      1) enable jemalloc background purging
      2) explicitly purge in flushdb
      09f99c2a
  12. 30 Sep, 2018 1 commit
    • Bruce Merry's avatar
      Fix incorrect memory usage accounting in zrealloc · 8fd1031b
      Bruce Merry authored
      When HAVE_MALLOC_SIZE is false, each call to zrealloc causes used_memory
      to increase by PREFIX_SIZE more than it should, due to mis-matched
      accounting between the original zmalloc (which includes PREFIX size in
      its increment) and zrealloc (which misses it from its decrement).
      
      I've also supplied a command-line test to easily demonstrate the
      problem. It's not wired into the test framework, because I don't know
      TCL so I'm not sure how to automate it.
      8fd1031b
  13. 16 Jul, 2018 1 commit
    • Oran Agra's avatar
      slave buffers were wasteful and incorrectly counted causing eviction · bf680b6f
      Oran Agra authored
      A) slave buffers didn't count internal fragmentation and sds unused space,
         this caused them to induce eviction although we didn't mean for it.
      
      B) slave buffers were consuming about twice the memory of what they actually needed.
      - this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time
        but networking.c not storing more than 16k (partially fixed recently in 237a38737).
      - besides it wasn't able to store half of the new string into one buffer and the
        other half into the next (so the above mentioned fix helped mainly for small items).
      - lastly, the sds buffers had up to 30% internal fragmentation that was wasted,
        consumed but not used.
      
      C) inefficient performance due to starting from a small string and reallocing many times.
      
      what i changed:
      - creating dedicated buffers for reply list, counting their size with zmalloc_size
      - when creating a new reply node from, preallocate it to at least 16k.
      - when appending a new reply to the buffer, first fill all the unused space of the
        previous node before starting a new one.
      
      other changes:
      - expose mem_not_counted_for_evict info field for the benefit of the test suite
      - add a test to make sure slave buffers are counted correctly and that they don't cause eviction
      bf680b6f
  14. 19 Jun, 2018 1 commit
    • Oran Agra's avatar
      add malloc_usable_size for libc malloc · 482785ac
      Oran Agra authored
      this reduces the extra 8 bytes we save before each pointer.
      but more importantly maybe, it makes the valgrind runs to be more similiar
      to our normal runs.
      
      note: the change in malloc_stats struct in server.h is to eliminate an name conflict.
      structs that are not typedefed are resolved from a separate name space.
      482785ac
  15. 12 Mar, 2018 1 commit
    • Oran Agra's avatar
      Adding real allocator fragmentation to INFO and MEMORY command + active defrag test · 806736cd
      Oran Agra authored
      other fixes / improvements:
      - LUA script memory isn't taken from zmalloc (taken from libc malloc)
        so it can cause high fragmentation ratio to be displayed (which is false)
      - there was a problem with "fragmentation" info being calculated from
        RSS and used_memory sampled at different times (now sampling them together)
      
      other details:
      - adding a few more allocator info fields to INFO and MEMORY commands
      - improve defrag test to measure defrag latency of big keys
      - increasing the accuracy of the defrag test (by looking at real grag info)
        this way we can use an even lower threshold and still avoid false positives
      - keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
      - add these the MEMORY DOCTOR command
      - deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
      - reduce sampling rate of the rss and allocator info
      806736cd
  16. 09 May, 2017 1 commit
  17. 10 Jan, 2017 1 commit
  18. 30 Dec, 2016 1 commit
  19. 19 Sep, 2016 1 commit
  20. 17 Dec, 2014 1 commit
  21. 12 Nov, 2014 1 commit
  22. 24 Mar, 2014 1 commit
    • antirez's avatar
      Sample and cache RSS in serverCron(). · 93253c27
      antirez authored
      Obtaining the RSS (Resident Set Size) info is slow in Linux and OSX.
      This slowed down the generation of the INFO 'memory' section.
      
      Since the RSS does not require to be a real-time measurement, we
      now sample it with server.hz frequency (10 times per second by default)
      and use this value both to show the INFO rss field and to compute the
      fragmentation ratio.
      
      Practically this does not make any difference for memory profiling of
      Redis but speeds up the INFO call significantly.
      93253c27
  23. 19 Nov, 2012 1 commit
  24. 24 Aug, 2012 1 commit
    • antirez's avatar
      Better Out of Memory handling. · 6fdc6354
      antirez authored
      The previous implementation of zmalloc.c was not able to handle out of
      memory in an application-specific way. It just logged an error on
      standard error, and aborted.
      
      The result was that in the case of an actual out of memory in Redis
      where malloc returned NULL (In Linux this actually happens under
      specific overcommit policy settings and/or with no or little swap
      configured) the error was not properly logged in the Redis log.
      
      This commit fixes this problem, fixing issue #509.
      Now the out of memory is properly reported in the Redis log and a stack
      trace is generated.
      
      The approach used is to provide a configurable out of memory handler
      to zmalloc (otherwise the default one logging the event on the
      standard output is used).
      6fdc6354
  25. 16 May, 2012 1 commit
  26. 05 Apr, 2012 1 commit
  27. 27 Mar, 2012 1 commit
  28. 07 Feb, 2012 1 commit
  29. 02 Jul, 2011 1 commit
  30. 20 Jun, 2011 1 commit
  31. 09 Jan, 2011 1 commit
  32. 02 Nov, 2010 1 commit
  33. 02 Sep, 2010 1 commit
  34. 24 Jul, 2010 1 commit
    • Benjamin Kramer's avatar
      Add zcalloc and use it where appropriate · 399f2f40
      Benjamin Kramer authored
      calloc is more effecient than malloc+memset when the system uses mmap to
      allocate memory. mmap always returns zeroed memory so the memset can be
      avoided.  The threshold to use mmap is 16k in osx libc and 128k in bsd
      libc and glibc. The kernel can lazily allocate the pages, this reduces
      memory usage when we have a page table or hash table that is mostly
      empty.
      
      This change is most visible when you start a new redis instance with vm
      enabled.  You'll see no increased memory usage no matter how big your
      page table is.
      399f2f40
  35. 01 Jul, 2010 1 commit
    • antirez's avatar
      redis.c split into many different C files. · e2641e09
      antirez authored
      networking related stuff moved into networking.c
      
      moved more code
      
      more work on layout of source code
      
      SDS instantaneuos memory saving. By Pieter and Salvatore at VMware ;)
      
      cleanly compiling again after the first split, now splitting it in more C files
      
      moving more things around... work in progress
      
      split replication code
      
      splitting more
      
      Sets split
      
      Hash split
      
      replication split
      
      even more splitting
      
      more splitting
      
      minor change
      e2641e09
  36. 19 Feb, 2010 1 commit
  37. 15 Jan, 2010 1 commit
  38. 23 Mar, 2009 1 commit
  39. 22 Mar, 2009 1 commit