1. 05 Feb, 2024 2 commits
    • guybe7's avatar
      Refactor the per-slot dict-array db.c into a new kvstore data structure (#12822) · 8cd62f82
      guybe7 authored
      # Description
      Gather most of the scattered `redisDb`-related code from the per-slot
      dict PR (#11695) and turn it to a new data structure, `kvstore`. i.e.
      it's a class that represents an array of dictionaries.
      
      # Motivation
      The main motivation is code cleanliness, the idea of using an array of
      dictionaries is very well-suited to becoming a self-contained data
      structure.
      This allowed cleaning some ugly code, among others: loops that run twice
      on the main dict and expires dict, and duplicate code for allocating and
      releasing this data structure.
      
      # Notes
      1. This PR reverts the part of https://github.com/redis/redis/pull/12848
      where the `rehashing` list is global (handling rehashing `dict`s is
      under the responsibility of `kvstore`, and should not be managed by the
      server)
      2. This PR also replaces the type of `server.pubsubshard_channels` from
      `dict**` to `kvstore` (original PR:
      https://github.com/redis/redis/pull/12804). After that was done,
      server.pubsub_channels was also chosen to be a `kvstore` (with only one
      `dict`, which seems odd) just to make the code cleaner by making it the
      same type as `server.pubsubshard_channels`, see
      `pubsubtype.serverPubSubChannels`
      3. the keys and expires kvstores are currenlty configured to allocate
      the individual dicts only when the first key is added (unlike before, in
      which they allocated them in advance), but they won't release them when
      the last key is deleted.
      
      Worth mentioning that due to the recent change the reply of DEBUG
      HTSTATS changed, in case no keys were ever added to the db.
      
      before:
      ```
      127.0.0.1:6379> DEBUG htstats 9
      [Dictionary HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      [Expires HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      ```
      
      after:
      ```
      127.0.0.1:6379> DEBUG htstats 9
      [Dictionary HT]
      [Expires HT]
      ```
      8cd62f82
    • Binbin's avatar
      Fix active expire timeout when db done the scanning (#13030) · f20774ec
      Binbin authored
      When db->expires_cursor==0, it means the DB is done the scanning,
      we should exit the loop to avoid the useless scanning.
      
      It is easy to see the active expire timeout in the modified test,
      for example, let's assume that there is only 1 expired key in the
      DB, and the size / buckets ratio is less than 1%, which means that
      we will skip it in isExpiryDictValidForSamplingCb, and the return
      value of expires_cursor is 0.
      
      Because `data.sampled == 0` is always true, so `repeat` is also
      always true, we will keep scanning the DB, but every time it is
      skipped by the previous judgment (expires_cursor = 0), until the
      timelimit is finally exhausted.
      f20774ec
  2. 31 Jan, 2024 3 commits
    • Binbin's avatar
      Fix dict resize allow test (#13016) · 9a7d3118
      Binbin authored
      Ci report this failure:
      ```
      *** [err]: Don't rehash if used memory exceeds maxmemory after rehash in tests/unit/maxmemory.tcl
      Expected '4098' to equal or match '4002'
      
      WARNING: the new maxmemory value set via CONFIG SET (1176088) is smaller than the current memory usage (1231083)
      ```
      
      It can be seen from the log that used_memory changed before we set
      maxmemory.
      The reason is that in #12819, in cron, in addition to trying to shrink,
      we will
      also tyring to expand. The dict was expanded by cron before we set
      maxmemory,
      causing the test to fail.
      
      Before setting maxmemory, we only add 4095 keys to avoid triggering
      resize.
      9a7d3118
    • Binbin's avatar
      Fix module assertion crash when timer and timeout are unlocked in the same event loop (#13015) · 6016973a
      Binbin authored
      When we use a timer to unblock a client in module, if the timer
      period and the block timeout are very close, they will unblock the
      client in the same event loop, and it will trigger the assertion.
      The reason is that in moduleBlockedClientTimedOut we will protect
      against re-processing, so we don't actually call updateStatsOnUnblock
      (see #12817), so we are not able to reset the c->duration. 
      
      The reason is unblockClientOnTimeout() didn't realize that bc had
      been unblocked. We add a function to the module to determine if bc
      is blocked, and then use it in unblockClientOnTimeout() to exit.
      
      There is the stack:
      ```
      beforeSleep
      blockedBeforeSleep
      handleBlockedClientsTimeout
      checkBlockedClientTimeout
      unblockClientOnTimeout
      unblockClient
      resetClient
      -- assertion, crash the server
      'c->duration == 0' is not true
      ```
      6016973a
    • Binbin's avatar
      Fix module unblock crash due to no timeout_callback (#13017) · 74a6e48a
      Binbin authored
      The block timeout is passed in the test case, but we do not pass
      in the timeout_callback, and it will crash when unlocking. In this
      case, in moduleBlockedClientTimedOut we will check timeout_callback.
      There is the stack:
      ```
      beforeSleep
      blockedBeforeSleep
      handleBlockedClientsTimeout
      checkBlockedClientTimeout
      unblockClientOnTimeout
      replyToBlockedClientTimedOut
      moduleBlockedClientTimedOut
      -- timeout_callback is NULL, invalidFunctionWasCalled
      bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
      ```
      74a6e48a
  3. 30 Jan, 2024 4 commits
    • Chen Tianjie's avatar
      Add novalues option to command HSCAN. (#12765) · f469dd8c
      Chen Tianjie authored
      
      
      Add a way to HSCAN a hash key, and get only the filed names.
      Command syntax is now:
      ```
      HSCAN key cursor [MATCH pattern] [COUNT count] [NOVALUES]
      ```
      when `NOVALUES` is on, the command will only return keys in the hash.
      
      ---------
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      f469dd8c
    • Slava Koyfman's avatar
      Implement `CLIENT KILL MAXAGE <maxage>` (#12299) · 24f6d08b
      Slava Koyfman authored
      
      
      Adds an ability to kill clients older than a specified age.
      
      Also, fixed the age calculation in `catClientInfoString` to use
      `commandTimeSnapshot`
      instead of the old `server.unixtime`, and added missing documentation
      for
      `CLIENT KILL ID` to output of `CLIENT help`.
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      24f6d08b
    • Oran Agra's avatar
      fix dict rehash tests introduced by #12802 broken by #12819 (#13009) · 7c9f41b5
      Oran Agra authored
      tests consistently fail on timeout (sleep that's too short).
      it now takes more time because in #12819 we iterate on all dicts, not
      just non-empty ones.
      it passed the PR's CI because it skips the `slow` tag, which might have
      been misplaced, but now it is probably required.
      with the fix, the tests take quite a lot of time:
      ```
      [ok]: Redis can trigger resizing (1860 ms)
      [ok]: Redis can rewind and trigger smaller slot resizing (744 ms)
      ```
      before #12819:
      ```
      [ok]: Redis can trigger resizing (309 ms)
      [ok]: Redis can rewind and trigger smaller slot resizing (295 ms)
      ```
      
      failure:
      https://github.com/redis/redis/actions/runs/7704158180/job/20995931735
      ```
      *** [err]: expire scan should skip dictionaries with lot's of empty buckets in tests/unit/expire.tcl
      scan didn't handle slot skipping logic.
      *** [err]: Redis can trigger resizing in tests/unit/other.tcl
      Expected '[Dictionary HT]
      Hash table 0 stats (main hash table):
       table size: 128
       number of elements: 5
      [Expires HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      ' to match '*table size: 8*' (context: type eval line 29 cmd {assert_match "*table size: 8*" [r debug HTSTATS 0]} proc ::test) 
      *** [err]: Redis can rewind and trigger smaller slot resizing in tests/unit/other.tcl
      Expected '[Dictionary HT]
      Hash table 0 stats (main hash table):
       table size: 256
       number of elements: 10
      [Expires HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      ' to match '*table size: 16*' (context: type eval line 27 cmd {assert_match "*table size: 16*" [r debug HTSTATS 0]} proc ::test) 
      ```
      7c9f41b5
    • Binbin's avatar
      Fix blocking commands timeout is reset due to re-processing command (#13004) · 492021db
      Binbin authored
      In #11012, we will reprocess command when client is unblocked on keys,
      in some blocking commands, for example, in the XREADGROUP BLOCK
      scenario,
      because of the re-processing command, we will recalculate the block
      timeout,
      causing the blocking time to be reset.
      
      This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
      let the command know that it is being re-processed, later in
      blockForKeys
      we will not reset the timeout.
      
      Affected BLOCK cases: 
      - list / zset / stream, added test cases for each.
      
      Unaffected cases:
      - module (never re-process the commands).
      - WAIT / WAITAOF (never re-process the commands).
      
      Fixes #12998.
      492021db
  4. 29 Jan, 2024 2 commits
    • Chen Tianjie's avatar
      Optimize resizing hash table to resize not only non-empty dicts. (#12819) · af7ceeb7
      Chen Tianjie authored
      The function `tryResizeHashTables` only attempts to shrink the dicts
      that has keys (change from #11695), this was a serious problem until the
      change in #12850 since it meant if all keys are deleted, we won't shrink
      the dick.
      But still, both dictShrink and dictExpand may be blocked by a fork child
      process, therefore, the cron job needs to perform both dictShrink and
      dictExpand, for not just non-empty dicts, but all dicts in DBs.
      
      What this PR does:
      
      1. Try to resize all dicts in DBs (not just non-empty ones, as it was
      since #12850)
      2. handle both shrink and expand (not just shrink, as it was since
      forever)
      3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
      `htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
      `dictExpandIfNeeded` which already contains all the code of those
      functions we get rid of, to make APIs more neat)
      4. In the `Don't rehash if redis has child process` test, now that cron
      would do resizing, we no longer need to write to DB after the child
      process got killed, and can wait for the cron to expand the hash table.
      af7ceeb7
    • Ozan Tezcan's avatar
      Add RM_TryCalloc() and RM_TryRealloc() (#12985) · c5273cae
      Ozan Tezcan authored
      Modules may want to handle allocation failures gracefully. Adding
      RM_TryCalloc() and RM_TryRealloc() for it.
      RM_TryAlloc() was added before:
      https://github.com/redis/redis/pull/10541
      c5273cae
  5. 27 Jan, 2024 1 commit
    • Roshan Khatri's avatar
      Reduce performance impact of dict rehashing and make it shorter. (#12899) · 5358bd7c
      Roshan Khatri authored
      
      
      #### Problem Statement:
      For any read/update operation during rehashing, we're doing ~10+ random
      DRAM lookups to do the rehashing, as we are using the `rehashidx` to
      rehash 10 buckets, whose dict entries most likely aren't cached in the
      CPU or near the bucket we are operating on. If these random bucket are
      empty, the rehashing process during that command execution is skipped.
      
      #### Implementation:
      For reducing the performance recession while dict is rehashing, we
      determine the index at which the key would be stored in the 0th HT, we
      check if that index has already been rehashed, if not we will rehash the
      bucket containing the key and the bucket will be moved from 0th HT to
      the 1st HT.
      
      If the key has already been rehashed, we perform the random access
      bucket rehash (using `rehashidx`) and we again verify if rehashing is
      still ongoing and look up the key in the respective HT.
      
      This ensures rehashing is not skipped in any command call and that we
      rehash a particular bucket or random bucket in each call.
      
      #### Changes in this PR:
      - Added a new method `dictBucketRehash` to perform rehash on a single
      bucket.
      - Helper function `moveKeysInBucketOldtoNew` for `dictRehash` and
      `dictBucketRehash` to move all the keys in a bucket from the old to the
      new hash HT.
      - Helper function `verifyMoreRehashRequired` for `dictRehash` and
      `dictBucketRehash` to check if we have already rehashed the whole table
      and if more rehashing is required.
      
      ### Benchmark:
      - This PR still shows **~13%** improvement in the latency during
      rehashing.
      
      - Rehashing is now **~2%** faster for this PR when compared to unstable.
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      5358bd7c
  6. 23 Jan, 2024 4 commits
    • Wen Hui's avatar
      Add INCR type command against wrong argument test cases. (#12836) · 68540913
      Wen Hui authored
      We have test cases for incr related commands with no key exist and
      spaces in key and wrong type of key. However, we dont have test cases
      covered for INCRBY INCRBYFLOAT DECRBY INCR DECR HINCRBY HINCRBYFLOAT
      ZINCRBY with valid key and invalid value as argument, and float value to
      incrby and decrby. So added test cases for the scenarios in incr.tcl.
      
      Thank you!
      68540913
    • Binbin's avatar
      Allow running WAITAOF in scripts, remove NOSCRIPT flag (#12977) · 85c31e0c
      Binbin authored
      In #11568 we removed the NOSCRIPT flag from commands, e.g. removing
      NOSCRIPT flag from WAIT. Aiming to allow them in scripts and let them
      implicitly behave in the non-blocking way.
      
      This PR remove NOSCRIPT flag from WAITAOF just like WAIT (to be
      symmetrical)).
      And this PR also add BLOCKING flag for WAIT and WAITAOF.
      85c31e0c
    • Binbin's avatar
      Some cleanups around function (#12940) · 628c0dea
      Binbin authored
      This PR did some cleanups around function:
      - drop the comment about Libraries Ctx, since we do have comment
        in functionsLibCtx, no need to maintain multiple copies.
      - remove outdated comment about the dropped Library description.
      - remove unused desc and code vars in functionExtractLibMetaData.
      - fix engines_nemory typo, changed it to engines_memory.
      - remove outdated comment about FUNCTION CREATE and FUNCTION INFO,
        FUNCTION CREATE was renamed to FUNCTION LOAD.
      - Check in initServer whether the return of functionsInit is OK.
      628c0dea
    • Harkrishn Patro's avatar
      Exit early if slowlog/acllog max len set to zero (#12965) · 2bce71b5
      Harkrishn Patro authored
      Currently slowlog gets disabled if slowlog-log-slower-than is set to less than zero. I think we should also disable it if slowlog-max-len is set to zero. We apply the same logic to acllog-max-len.
      2bce71b5
  7. 19 Jan, 2024 1 commit
    • Yanqi Lv's avatar
      Change the threshold of dict expand, shrink and rehash (#12948) · b07174af
      Yanqi Lv authored
      Before this change (most recently modified in
      https://github.com/redis/redis/pull/12850#discussion_r1421406393), The
      trigger for normal expand threshold was 100% utilization and the trigger
      for normal shrink threshold was 10% (HASHTABLE_MIN_FILL).
      While during fork (DICT_RESIZE_AVOID), when we want to avoid rehash, the
      trigger thresholds were multiplied by 5 (`dict_force_resize_ratio`),
      meaning 500% for expand and 2% (100/10/5) for shrink.
      
      However, in `dictRehash` (the incremental rehashing), the rehashing
      threshold for shrinking during fork (DICT_RESIZE_AVOID) was 20% by
      mistake.
      This meant that if a shrinking is triggered when `dict_can_resize` is
      `DICT_RESIZE_ENABLE` which the threshold is 10%, the rehashing can
      continue when `dict_can_resize` is `DICT_RESIZE_AVOID`.
      This would cause unwanted CopyOnWrite damage.
      
      It'll make sense to change the thresholds of the rehash trigger and the
      thresholds of the incremental rehashing the same, however, in one we
      compare the size of the hash table to the number of records, and in the
      other we compare the size of ht[0] to the size of ht[1], so the formula
      is not exactly the same.
      
      to make things easier we change all the thresholds to powers of 2, so
      the normal shrinking threshold is changed from 100/10 (i.e. 10%) to
      100/8 (i.e. 12.5%), and we change the threshold during forks from 5 to
      4, i.e. from 500% to 400% for expand, and from 2% (100/10/5) to 3.125%
      (100/8/4)
      b07174af
  8. 18 Jan, 2024 1 commit
    • Binbin's avatar
      Fix unexpected resize causing test failure (#12960) · 29e6245a
      Binbin authored
      Before #12850, we will only try to shrink the dict in serverCron,
      which we can control by using a child process, but now every time
      we delete a key, the shrink check will be called.
      
      In these test (added in #12802), we meant to disable the resizing,
      but druing the delete, the dict will meet the force shrink, like
      2 / 128 = 0.015 < 0.2, the delete will trigger a force resize and
      will cause the test to fail.
      
      In this commit, we try to keep the load factor at 3 / 128 = 0.023,
      that is, do not meet the force shrink.
      29e6245a
  9. 17 Jan, 2024 1 commit
    • Binbin's avatar
      Fix race in slot dict resize test (#12942) · 131d95f2
      Binbin authored
      The test have a race:
      ```
      *** [err]: Redis can rewind and trigger smaller slot resizing in tests/unit/other.tcl
      Expected '[Dictionary HT]
      Hash table 0 stats (main hash table):
       table size: 12
       number of elements: 2
      [Expires HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      ' to match '*table size: 8*' (context: type eval line 12 cmd {assert_match "*table size: 8*" [r debug HTSTATS 0]} proc ::test)
      ```
      
      When `r del "{alice}$j"` is executed in the loop, when the key is
      deleted to [9, 12], the load factor has meet HASHTABLE_MIN_FILL,
      if serverCron happens to trigger slot dict resize, then the test
      will fail. Because there is not way to meet HASHTABLE_MIN_FILL in
      the subsequent dels.
      
      The solution is to avoid triggering the resize in advance. We can
      use multi to delete them at once, or we can disable the resize.
      Since we disabled resize in the previous test, the fix also uses
      the method of disabling resize.
      
      The test is introduced in #12802.
      131d95f2
  10. 15 Jan, 2024 2 commits
    • Yanqi Lv's avatar
      Shrink dict when deleting dictEntry (#12850) · e2b7932b
      Yanqi Lv authored
      When we insert entries into dict, it may autonomously expand if needed.
      However, when we delete entries from dict, it doesn't shrink to the
      proper size. If there are few entries in a very large dict, it may cause
      huge waste of memory and inefficiency when iterating.
      
      The main keyspace dicts (keys and expires), are shrinked by cron
      (`tryResizeHashTables` calls `htNeedsResize` and `dictResize`),
      And some data structures such as zset and hash also do that (call
      `htNeedsResize`) right after a loop of calls to `dictDelete`,
      But many other dicts are completely missing that call (they can only
      expand).
      
      In this PR, we provide the ability to automatically shrink the dict when
      deleting. The conditions triggering the shrinking is the same as
      `htNeedsResize` used to have. i.e. we expand when we're over 100%
      utilization, and shrink when we're below 10% utilization.
      
      Additionally:
      * Add `dictPauseAutoResize` so that flows that do mass deletions, will
      only trigger shrinkage at the end.
      * Rename `dictResize` to `dictShrinkToFit` (same logic as it used to
      have, but better name describing it)
      * Rename `_dictExpand` to `_dictResize` (same logic as it used to have,
      but better name describing it)
       
      related to discussion
      https://github.com/redis/redis/pull/12819#discussion_r1409293878
      
      
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarzhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
      e2b7932b
    • zhaozhao.zz's avatar
      fix scripts access wrong slot if they disagree with pre-declared keys (#12906) · bb2b6e29
      zhaozhao.zz authored
      Regarding how to obtain the hash slot of a key, there is an optimization
      in `getKeySlot()`, it is used to avoid redundant hash calculations for
      keys: when the current client is in the process of executing a command,
      it can directly use the slot of the current client because the slot to
      access has already been calculated in advance in `processCommand()`.
      
      However, scripts are a special case where, in default mode or with
      `allow-cross-slot-keys` enabled, they are allowed to access keys beyond
      the pre-declared range. This means that the keys they operate on may not
      belong to the slot of the pre-declared keys. Currently, when the
      commands in a script are executed, the slot of the original client
      (i.e., the current client) is not correctly updated, leading to
      subsequent access to the wrong slot.
      
      This PR fixes the above issue. When checking the cluster constraints in
      a script, the slot to be accessed by the current command is set for the
      original client (i.e., the current client). This ensures that
      `getKeySlot()` gets the correct slot cache.
      
      Additionally, the following modifications are made:
      
      1. The 'sort' and 'sort_ro' commands use `getKeySlot()` instead of
      `c->slot` because the client could be an engine client in a script and
      can lead to potential bug.
      2. `getKeySlot()` is also used in pubsub to obtain the slot for the
      channel, standardizing the way slots are retrieved.
      bb2b6e29
  11. 11 Jan, 2024 1 commit
  12. 10 Jan, 2024 1 commit
  13. 09 Jan, 2024 1 commit
  14. 03 Jan, 2024 1 commit
    • Madelyn Olson's avatar
      Handle recursive serverAsserts and provide more information for recursive segfaults (#12857) · 068051e3
      Madelyn Olson authored
      This change is trying to make two failure modes a bit easier to deep dive:
      1. If a serverPanic or serverAssert occurs during the info (or module)
      printing, it will recursively panic, which is a lot of fun as it will
      just keep recursively printing. It will eventually stack overflow, but
      will generate a lot of text in the process.
      2. When a segfault happens during the segfault handler, no information
      is communicated other than it happened. This can be problematic because
      `info` may help diagnose the real issue, but without fixing the
      recursive crash it might be hard to get at that info.
      068051e3
  15. 27 Dec, 2023 1 commit
    • sundb's avatar
      Fix oom-score-adj test due to no permission (#12887) · bef57153
      sundb authored
      
      
      Fix #12792
      
      On ubuntu 23(lunar), non-root users will not be allowed to change the
      oom_score_adj of a process to a value that is too low.
      Since terminal's default oom_score_adj is 200, if we run the test on
      terminal, we won't be able to set the oom_score_adj of the redis process
      to 9 or 22, which is too low.
      
      Reproduction on ubuntu 23(lunar) terminal:
      ```sh
      $ cat /proc/`pgrep redis-server`/oom_score_adj
      200
      $ echo 100 > /proc/`pgrep redis-server`/oom_score_adj
      # success without error
      $ echo 99 > /proc/`pgrep redis-server`/oom_score_adj
      echo: write error: Permission denied
      ```
      
      As from the output above, we can only set the minimum oom score of redis
      processes to 100.
      By modifying the test, make oom_score_adj only increase upwards and not
      decrease.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      bef57153
  16. 24 Dec, 2023 2 commits
  17. 17 Dec, 2023 1 commit
  18. 13 Dec, 2023 3 commits
    • Chen Tianjie's avatar
      Support by/get options for sort(_ro) in cluster mode when pattern implies slot. (#12728) · e95a5d48
      Chen Tianjie authored
      The by/get options of sort/sort_ro command used to be forbidden in
      cluster mode, since we are not sure which slot the pattern may be in.
      
      As the optimization done in #12536, patterns now can be mapped to slots,
      we should allow by/get options in cluster mode when the pattern maps to
      the same slot as the key.
      e95a5d48
    • Binbin's avatar
      Redact ACL username information and mark *-key-file-pass configs as sensitive (#12860) · 3c0fd252
      Binbin authored
      In #11489, we consider acl username to be sensitive information,
      and consider the ACL GETUSER a sensitive command and remove it
      from redis-cli historyfile.
      
      This PR redact username information in ACL GETUSER and ACL DELUSER
      from SLOWLOG, and also remove ACL DELUSER from redis-cli historyfile.
      
      This PR also mark tls-key-file-pass and tls-client-key-file-pass
      as sensitive config, will redact it from SLOWLOG and also
      remove them from redis-cli historyfile.
      3c0fd252
    • Chen Tianjie's avatar
      Add metric to INFO CLIENTS: pubsub_clients. (#12849) · f9cc25c1
      Chen Tianjie authored
      In INFO CLIENTS section, we already have blocked_clients and
      tracking_clients. We should add a new metric showing the number of
      pubsub connections, which helps performance monitoring and trouble
      shooting.
      f9cc25c1
  19. 11 Dec, 2023 1 commit
    • Binbin's avatar
      Fix delKeysInSlot server events are not executed inside an execution unit (#12745) · c85a9b78
      Binbin authored
      This is a follow-up fix to #12733. We need to apply the same changes to
      delKeysInSlot. Refer to #12733 for more details.
      
      This PR contains some other minor cleanups / improvements to the test
      suite and docs.
      It uses the postnotifications test module in a cluster mode test which
      revealed a leak in the test module (fixed).
      c85a9b78
  20. 05 Dec, 2023 1 commit
  21. 29 Nov, 2023 1 commit
    • zhaozhao.zz's avatar
      format cpu config as redis style (#7351) · 3431b1f1
      zhaozhao.zz authored
      The following four configurations are renamed to align with Redis style:
      
      1. server_cpulist renamed to server-cpulist
      2. bio_cpulist renamed to bio-cpulist
      3. aof_rewrite_cpulist renamed to aof-rewrite-cpulist
      4. bgsave_cpulist renamed to bgsave-cpulist
      
      The original names are retained as aliases to ensure compatibility with
      old configuration files. We recommend users to gradually transition to
      using the new configuration names to maintain consistency in style.
      3431b1f1
  22. 28 Nov, 2023 1 commit
    • zhaozhao.zz's avatar
      Fix resize hash tables stuck on the last non-empty slot (#12802) · a1c5171c
      zhaozhao.zz authored
      Introduced in #11695 .
      
      The tryResizeHashTables function gets stuck on the last non-empty slot
      while iterating through dictionaries. It does not restart from the
      beginning. The reason for this issue is a problem with the usage of
      dbIteratorNextDict:
      
      /* Returns next dictionary from the iterator, or NULL if iteration is complete. */
      dict *dbIteratorNextDict(dbIterator *dbit) {
          if (dbit->next_slot == -1) return NULL;
          dbit->slot = dbit->next_slot;
          dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
          return dbGetDictFromIterator(dbit);
      }
      
      When iterating to the last non-empty slot, next_slot is set to -1,
      causing it to loop indefinitely on that slot. We need to modify the code
      to ensure that after iterating to the last non-empty slot, it returns to
      the first non-empty slot.
      
      BTW, function tryResizeHashTables is actually iterating over slots
      that have keys. However, in its implementation, it leverages the
      dbIterator (which is a key iterator) to obtain slot and dictionary
      information. While this approach works fine, but it is not very
      intuitive. This PR also improves readability by changing the iteration
      to directly iterate over slots, thereby enhancing clarity.
      a1c5171c
  23. 27 Nov, 2023 1 commit
    • Binbin's avatar
      Un-register notification and server event when RedisModule_OnLoad fails (#12809) · d6f19539
      Binbin authored
      When we register notification or server event in RedisModule_OnLoad, but
      RedisModule_OnLoad eventually fails, triggering notification or server
      event
      will cause the server to crash.
      
      If the loading fails on a later stage of moduleLoad, we do call
      moduleUnload
      which handles all un-registration, but when it fails on the
      RedisModule_OnLoad
      call, we only un-register several specific things and these were
      missing:
      
      - moduleUnsubscribeNotifications
      - moduleUnregisterFilters
      - moduleUnsubscribeAllServerEvents
      
      Refactored the code to reuse the code from moduleUnload.
      
      Fixes #12808.
      d6f19539
  24. 23 Nov, 2023 1 commit
    • meiravgri's avatar
      Fix async safety in signal handlers (#12658) · 2e854bcc
      meiravgri authored
      see discussion from after https://github.com/redis/redis/pull/12453 was
      merged
      ----
      This PR replaces signals that are not considered async-signal-safe
      (AS-safe) with safe calls.
      
      #### **1. serverLog() and serverLogFromHandler()**
      `serverLog` uses unsafe calls. It was decided that we will **avoid**
      `serverLog` calls by the signal handlers when:
      * The signal is not fatal, such as SIGALRM. In these cases, we prefer
      using `serverLogFromHandler` which is the safe version of `serverLog`.
      Note they have different prompts:
      `serverLog`: `62220:M 26 Oct 2023 14:39:04.526 # <msg>`
      `serverLogFromHandler`: `62220:signal-handler (1698331136) <msg>`
      * The code was added recently. Calls to `serverLog` by the signal
      handler have been there ever since Redis exists and it hasn't caused
      problems so far. To avoid regression, from now we should use
      `serverLogFromHandler`
      
      #### **2. `snprintf` `fgets` and `strtoul`(base = 16) -------->
      `_safe_snprintf`, `fgets_async_signal_safe`, `string_to_hex`**
      The safe version of `snprintf` was taken from
      [here](https://github.com/twitter/twemcache/blob/8cfc4ca5e76ed936bd3786c8cc43ed47e7778c08/src/mc_util.c#L754)
      
      #### **3. fopen(), fgets(), fclose() --------> open(), read(), close()**
      
      #### **4. opendir(), readdir(), closedir() --------> open(),
      syscall(SYS_getdents64), close()**
      
      #### **5. Threads_mngr sync mechanisms**
      * waiting for the thread to generate stack trace: semaphore -------->
      busy-wait
      * `globals_rw_lock` was removed: as we are not using malloc and the
      semaphore anymore we don't need to protect `ThreadsManager_cleanups`.
      
      #### **6. Stacktraces buffer**
      The initial problem was that we were not able to safely call malloc
      within the signal handler.
      To solve that we created a buffer on the stack of `writeStacktraces` and
      saved it in a global pointer, assuming that under normal circumstances,
      the function `writeStacktraces` would complete before any thread
      attempted to write to it. However, **if threads lag behind, they might
      access this global pointer after it no longer belongs to the
      `writeStacktraces` stack, potentially corrupting memory.**
      To address this, various solutions were discussed
      [here](https://github.com/redis/redis/pull/12658#discussion_r1390442896)
      Eventually, we decided to **create a pipe** at server startup that will
      remain valid as long as the process is alive.
      We chose this solution due to its minimal memory usage, and since
      `write()` and `read()` are atomic operations. It ensures that stack
      traces from different threads won't mix.
      
      **The stacktraces collection process is now as  follows:**
      * Cleaning the pipe to eliminate writes of late threads from previous
      runs.
      * Each thread writes to the pipe its stacktrace
      * Waiting for all the threads to mark completion or until a timeout (2
      sec) is reached
      * Reading from the pipe to print the stacktraces.
      
      #### **7. Changes that were considered and eventually were dropped**
      * replace watchdog timer with a POSIX timer: 
      according to [settimer man](https://linux.die.net/man/2/setitimer)
      
      > POSIX.1-2008 marks getitimer() and setitimer() obsolete, recommending
      the use of the POSIX timers API
      ([timer_gettime](https://linux.die.net/man/2/timer_gettime)(2),
      [timer_settime](https://linux.die.net/man/2/timer_settime)(2), etc.)
      instead.
      
      However, although it is supposed to conform to POSIX std, POSIX timers
      API is not supported on Mac.
      You can take a look here at the Linux implementation:
      
      [here](https://github.com/redis/redis/commit/c7562ee13546e504977372fdf40d33c3f86775a5
      
      )
      To avoid messing up the code, and uncertainty regarding compatibility,
      it was decided to drop it for now.
      
      * avoid using sds (uses malloc) in logConfigDebugInfo
      It was considered to print config info instead of using sds, however
      apparently, `logConfigDebugInfo` does more than just print the sds, so
      it was decided this fix is out of this issue scope.
      
      #### **8. fix Signal mask check**
      The check `signum & sig_mask` intended to indicate whether the signal is
      blocked by the thread was incorrect. Actually, the bit position in the
      signal mask corresponds to the signal number. We fixed this by changing
      the condition to: `sig_mask & (1L << (sig_num - 1))`
      
      #### **9. Unrelated changes**
      both `fork.tcl `and `util.tcl` implemented a function called
      `count_log_message` expecting different parameters. This caused
      confusion when trying to run daily tests with additional test parameters
      to run a specific test.
      The `count_log_message` in `fork.tcl` was removed and the calls were
      replaced with calls to `count_log_message` located in `util.tcl`
      
      ---------
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2e854bcc
  25. 19 Nov, 2023 1 commit
  26. 14 Nov, 2023 1 commit
    • Binbin's avatar
      Fix DB iterator not resetting pauserehash causing dict being unable to rehash (#12757) · fe363063
      Binbin authored
      When using DB iterator, it will use dictInitSafeIterator to init a old safe
      dict iterator. When dbIteratorNext is used, it will jump to the next slot db
      dict when we are done a dict. During this process, we do not have any calls to
      dictResumeRehashing, which causes the dict's pauserehash to always be > 0.
      
      And at last, it will be returned directly in dictRehashMilliseconds, which causes
      us to have slot dict in a state where rehash cannot be completed.
      
      In the "expire scan should skip dictionaries with lot's of empty buckets" test,
      adding a `keys *` can reproduce the problem stably. `keys *` will call dbIteratorNext
      to trigger a traversal of all slot dicts.
      
      Added dbReleaseIterator and dbIteratorInitNextSafeIterator methods to call dictResetIterator.
      Issue was introduced in #11695.
      fe363063