1. 20 May, 2020 1 commit
    • Oran Agra's avatar
      fix a rare active defrag edge case bug leading to stagnation · 88d71f47
      Oran Agra authored
      There's a rare case which leads to stagnation in the defragger, causing
      it to keep scanning the keyspace and do nothing (not moving any
      allocation), this happens when all the allocator slabs of a certain bin
      have the same % utilization, but the slab from which new allocations are
      made have a lower utilization.
      
      this commit fixes it by removing the current slab from the overall
      average utilization of the bin, and also eliminate any precision loss in
      the utilization calculation and move the decision about the defrag to
      reside inside jemalloc.
      
      and also add a test that consistently reproduce this issue.
      88d71f47
  2. 18 Feb, 2020 1 commit
    • Oran Agra's avatar
      Defrag big lists in portions to avoid latency and freeze · 485425ce
      Oran Agra authored
      When active defrag kicks in and finds a big list, it will create a bookmark to
      a node so that it is able to resume iteration from that node later.
      
      The quicklist manages that bookmark, and updates it in case that node is deleted.
      
      This will increase memory usage only on lists of over 1000 (see
      active-defrag-max-scan-fields) quicklist nodes (1000 ziplists, not 1000 items)
      by 16 bytes.
      
      In 32 bit build, this change reduces the maximum effective config of
      list-compress-depth and list-max-ziplist-size (from 32767 to 8191)
      485425ce
  3. 12 Nov, 2019 1 commit
    • Oran Agra's avatar
      Adjustments for active defrag defaults and tuning · 0bc3dab0
      Oran Agra authored
      Reduce default minimum effort, so that when fragmentation is just detected,
      the impact on the latency will be minor.
      
      Reduce the default maximum effort, mainly to prevent a case were a sudden
      massive deletions, won't trigger an aggressive defrag that will cause latency.
      
      When activedefrag is disabled mid-run, reset the 'running' info field, and
      clear the scan cursor, so that when it'll be re-enabled, a new fresh scan will
      start.
      
      Clearing the 'running' variable is important since lowering the defragger
      tunables mid-scan won't help, the defragger only considers new threshold when
      a new scan starts, and during a scan it can only become more aggressive,
      (when more severe fragmentation is detected), it'll never go less aggressive.
      So by temporarily disabling activedefrag, one can lower th the tunables.
      
      Removing the experimantal warning.
      0bc3dab0
  4. 07 Oct, 2019 1 commit
    • Oran Agra's avatar
      fix issues found by a static analyzer · d1a005ab
      Oran Agra authored
      cluster.c - stack buffer memory alignment
          The pointer 'buf' is cast to a more strictly aligned pointer type
      evict.c - lazyfree_lazy_eviction, lazyfree_lazy_eviction always called
      defrag.c - bug in dead code
      server.c - casting was missing parenthesis
      rax.c - indentation / newline suggested an 'else if' was intended
      d1a005ab
  5. 27 Sep, 2019 1 commit
  6. 17 Jul, 2019 1 commit
    • Oran Agra's avatar
      Module API for Forking · 56258c6b
      Oran Agra authored
      * create module API for forking child processes.
      * refactor duplicate code around creating and tracking forks by AOF and RDB.
      * child processes listen to SIGUSR1 and dies exitFromChild in order to
        eliminate a valgrind warning of unhandled signal.
      * note that BGSAVE error reply has changed.
      
      valgrind error is:
        Process terminating with default action of signal 10 (SIGUSR1)
      56258c6b
  7. 08 May, 2019 1 commit
  8. 18 Jul, 2018 1 commit
    • Oran Agra's avatar
      make active defrag test more stable · f89c93c8
      Oran Agra authored
      on slower machines, the active defrag test tended to fail.
      although the fragmentation ratio was below the treshold, the defragger was
      still in the middle of a scan cycle.
      
      this commit changes:
      - the defragger uses the current fragmentation state, rather than the cache one
        that is updated by server cron every 100ms. this actually fixes a bug of
        starting one excess scan cycle
      - the test lets the defragger use more CPU cycles, in hope that the defrag
        will be faster, but also give it more time before we give up.
      f89c93c8
  9. 27 Jun, 2018 1 commit
  10. 12 Mar, 2018 2 commits
    • Oran Agra's avatar
      Adding real allocator fragmentation to INFO and MEMORY command + active defrag test · 806736cd
      Oran Agra authored
      other fixes / improvements:
      - LUA script memory isn't taken from zmalloc (taken from libc malloc)
        so it can cause high fragmentation ratio to be displayed (which is false)
      - there was a problem with "fragmentation" info being calculated from
        RSS and used_memory sampled at different times (now sampling them together)
      
      other details:
      - adding a few more allocator info fields to INFO and MEMORY commands
      - improve defrag test to measure defrag latency of big keys
      - increasing the accuracy of the defrag test (by looking at real grag info)
        this way we can use an even lower threshold and still avoid false positives
      - keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
      - add these the MEMORY DOCTOR command
      - deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
      - reduce sampling rate of the rss and allocator info
      806736cd
    • Oran Agra's avatar
      active defrag v2 · be1b4aa9
      Oran Agra authored
      - big keys are not defragged in one go from within the dict scan
        instead they are scanned in parts after the main dict hash bucket is done.
      - add latency monitor sample for defrag
      - change default active-defrag-cycle-min to induce lower latency
      - make active defrag start a new scan right away if needed, so it's easier
        (for the test suite) to detect when it's done
      - make active defrag quick the current cycle after each db / big key
      - defrag  some non key long term global allocations
      - some refactoring for smaller functions and more reusable code
      - during dict rehashing, one scan iteration of the dict, can end up scanning
        one bucket in the smaller dict and many many buckets in the larger dict.
        so waiting for 16 scan iterations before checking the time, may be much too long.
      be1b4aa9
  11. 05 Dec, 2017 1 commit
  12. 12 Jan, 2017 1 commit
  13. 11 Jan, 2017 2 commits
  14. 10 Jan, 2017 2 commits
  15. 02 Jan, 2017 1 commit
  16. 30 Dec, 2016 1 commit