1. 14 Nov, 2023 1 commit
    • Binbin's avatar
      Fix DB iterator not resetting pauserehash causing dict being unable to rehash (#12757) · fe363063
      Binbin authored
      When using DB iterator, it will use dictInitSafeIterator to init a old safe
      dict iterator. When dbIteratorNext is used, it will jump to the next slot db
      dict when we are done a dict. During this process, we do not have any calls to
      dictResumeRehashing, which causes the dict's pauserehash to always be > 0.
      
      And at last, it will be returned directly in dictRehashMilliseconds, which causes
      us to have slot dict in a state where rehash cannot be completed.
      
      In the "expire scan should skip dictionaries with lot's of empty buckets" test,
      adding a `keys *` can reproduce the problem stably. `keys *` will call dbIteratorNext
      to trigger a traversal of all slot dicts.
      
      Added dbReleaseIterator and dbIteratorInitNextSafeIterator methods to call dictResetIterator.
      Issue was introduced in #11695.
      fe363063
  2. 13 Nov, 2023 1 commit
  3. 12 Nov, 2023 1 commit
    • Roshan Khatri's avatar
      Add DEBUG_ASSERTIONS option to custom assert (#12667) · 88e83e51
      Roshan Khatri authored
      This PR introduces a new macro, serverAssertWithInfoDebug, to do complex assertions only for debugging. The main intention is to allow running complex operations during tests without impacting runtime performance. This assertion is enabled when setting DEBUG_ASSERTIONS.
      
      The DEBUG_ASSERTIONS flag is set for the daily and CI variants of `test-sanitizer-address`.
      88e83e51
  4. 11 Nov, 2023 1 commit
    • Harkrishn Patro's avatar
      Increase timeout for expiry cluster tests (#12752) · 9ca84903
      Harkrishn Patro authored
      Test recently added fails on timeout in valgrind in GH actions.
      
      Locally with valgrind the test finishes within 1.5 sec(s). Couldn't find
      any issue due to lack of reproducibility. Increasing the timeout and
      adding an additional log to the test to understand how many keys
      were left at the end.
      9ca84903
  5. 10 Nov, 2023 1 commit
  6. 09 Nov, 2023 1 commit
  7. 08 Nov, 2023 2 commits
    • Binbin's avatar
      Fix genClusterDebugString minor sds leaks (#12739) · 53294e53
      Binbin authored
      This function now will only be called in printCrashReport,
      so this is just a cleanup.
      53294e53
    • Meir Shpilraien (Spielrein)'s avatar
      Before evicted and before expired server events are not executed inside an execution unit. (#12733) · 0ffb9d2e
      Meir Shpilraien (Spielrein) authored
      Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`.
      This new event allows the module to read the key data just before it is removed
      from the database (either deleted, expired, evicted, or overwritten).
      
      When the key is removed from the database, either by active expire or eviction.
      The new event was not called as part of an execution unit. This can cause an
      issue if the module registers a post notification job inside the event. This job will
      not be executed atomically with the expiration/eviction operation and will not
      replicated inside a Multi/Exec. Moreover, the post notification job will be executed
      right after the event where it is still not safe to perform any write operation, this will
      violate the promise that post notification job will be called atomically with the
      operation that triggered it and **only when it is safe to write**.
      
      This PR fixes the issue by wrapping each expiration/eviction of a key with an execution
      unit. This makes sure the entire operation will run atomically and all the post notification
      jobs will be executed at the end where it is safe to write.
      
      Tests were modified to verify the fix.
      0ffb9d2e
  8. 06 Nov, 2023 3 commits
  9. 05 Nov, 2023 1 commit
    • Wen Hui's avatar
      Fix the bug that write redis sensitive command information to redis_cli historyfile (#11489) · 28b6155b
      Wen Hui authored
      Currently, we do not write the following sensitive commands into the ~/.rediscli_history file:
      
      ACL SETUSER username [rule [rule ...]]
      AUTH [username] password
      HELLO [AUTH username password] 
      MIGRATE host port <key | ""> destination-db timeout [[AUTH password | AUTH2 username password]]
      CONFIG SET masterauth master-password
      CONFIG SET masteruser username
      CONFIG SET requirepass foobared
      
      However, we still write the following sensitive commands into the ~/.rediscli_history file:
      ACL GETUSER username
      Sentinel CONFIG set sentinel-pass password
      Sentinel CONFIG set sentinel-user username
      Sentinel set mastername auth-pass password
      Sentinel set mastername auth-user username
      
      This change adds the commands of the second list to be skipped from being written to the history file.
      
      28b6155b
  10. 02 Nov, 2023 2 commits
  11. 01 Nov, 2023 2 commits
  12. 31 Oct, 2023 2 commits
  13. 28 Oct, 2023 1 commit
  14. 27 Oct, 2023 2 commits
    • Harkrishn Patro's avatar
      Reduce dbBuckets operation time complexity from O(N) to O(1) (#12697) · 4145d628
      Harkrishn Patro authored
      
      
      As part of #11695 independent dictionaries were introduced per slot.
      Time complexity to discover total no. of buckets across all dictionaries
      increased to O(N) with straightforward implementation of iterating over
      all dictionaries and adding dictBuckets of each.
      
      To optimize the time complexity, we could maintain a global counter at
      db level to keep track of the count of buckets and update it on the start
      and end of rehashing.
      
      ---------
      Co-authored-by: default avatarRoshan Khatri <rvkhatri@amazon.com>
      4145d628
    • Roshan Khatri's avatar
      Reset later item flag after defrag later is done (#12694) · 7d68208a
      Roshan Khatri authored
      Fixing issues described in #12672, started after #11695
      Related to #12674
      
      Fixes the `defrag didn't stop' issue.
      
      In some cases of how the keys were stored in memory
      defrag_later_item_in_progress was not getting reset once we finish
      defragging the later items and we move to the next slot. This stopped
      the scan to happen in the later slots and did not get 
      7d68208a
  15. 25 Oct, 2023 1 commit
  16. 24 Oct, 2023 2 commits
  17. 22 Oct, 2023 1 commit
    • Harkrishn Patro's avatar
      Fix defrag test (#12674) · 26eb4ce3
      Harkrishn Patro authored
      Fixing issues started after #11695 when the defrag tests are being executed in cluster mode too.
      For some reason, it looks like the defragmentation is over too quickly, before the test is able to
      detect that it's running.
      so now instead of waiting to see that it's active, we wait to see that it did some work
      ```
      [err]: Active defrag big list: cluster in tests/unit/memefficiency.tcl
      defrag not started.
      [err]: Active defrag big keys: cluster in tests/unit/memefficiency.tcl
      defrag didn't stop.
      ```
      26eb4ce3
  18. 19 Oct, 2023 2 commits
    • Harkrishn Patro's avatar
      Disable flaky defrag tests affecting daily run (#12672) · becd50d0
      Harkrishn Patro authored
      Temporarily disabling few of the defrag tests in cluster mode to make the daily run stable:
      
      Active defrag eval scripts
      Active defrag big keys
      Active defrag big list
      Active defrag edge case
      becd50d0
    • Harkrishn Patro's avatar
      Fix resize hash table dictionary iterator (#12660) · f3bf8485
      Harkrishn Patro authored
      Dictionary iterator logic in the `tryResizeHashTables` method is picking the next
      (incorrect) dictionary while the cursor is at a given slot. This could lead to some
      dictionary/slot getting skipped from resizing.
      
      Also stabilize the test.
      
      problem introduced recently in #11695
      f3bf8485
  19. 18 Oct, 2023 2 commits
  20. 16 Oct, 2023 1 commit
    • meiravgri's avatar
      remove heap allocations from signal handlers. (#12655) · d27c7413
      meiravgri authored
      Using heap allocation during signal handlers is unsafe.
      This PR purpose is to replace all the heap allocations done within the signal
      handlers raised upon server crash and assertions.
      These were added in #12453.
      
      writeStacktraces(): allocates the stacktraces output array on the calling thread's
      stack and assigns the address to a global variable.
      It calls `ThreadsManager_runOnThreads()` that invokes `collect_stacktrace_data()`
      by each thread: each thread writes to a different location in the above array to allow
      sync writes.
      
      get_ready_to_signal_threads_tids(): instead of allocating the `tids` array, it receives it
      as a fixed size array parameter, allocated on on the stack of the calling function, and
      returns the number of valid threads. The array size is hard-coded to 50.
      
      `ThreadsManager_runOnThreads():` To avoid the outputs array allocation, the
      **callback signature** was changed. Now it should return void. This function return type
      has also changed to int - returns 1 if successful, and 0 otherwise.
      
      Other unsafe calls will be handled in following PRs
      d27c7413
  21. 15 Oct, 2023 1 commit
    • Vitaly's avatar
      Replace cluster metadata with slot specific dictionaries (#11695) · 0270abda
      Vitaly authored
      This is an implementation of https://github.com/redis/redis/issues/10589
      
       that eliminates 16 bytes per entry in cluster mode, that are currently used to create a linked list between entries in the same slot.  Main idea is splitting main dictionary into 16k smaller dictionaries (one per slot), so we can perform all slot specific operations, such as iteration, without any additional info in the `dictEntry`. For Redis cluster, the expectation is that there will be a larger number of keys, so the fixed overhead of 16k dictionaries will be The expire dictionary is also split up so that each slot is logically decoupled, so that in subsequent revisions we will be able to atomically flush a slot of data.
      
      ## Important changes
      * Incremental rehashing - one big change here is that it's not one, but rather up to 16k dictionaries that can be rehashing at the same time, in order to keep track of them, we introduce a separate queue for dictionaries that are rehashing. Also instead of rehashing a single dictionary, cron job will now try to rehash as many as it can in 1ms.
      * getRandomKey - now needs to not only select a random key, from the random bucket, but also needs to select a random dictionary. Fairness is a major concern here, as it's possible that keys can be unevenly distributed across the slots. In order to address this search we introduced binary index tree). With that data structure we are able to efficiently find a random slot using binary search in O(log^2(slot count)) time.
      * Iteration efficiency - when iterating dictionary with a lot of empty slots, we want to skip them efficiently. We can do this using same binary index that is used for random key selection, this index allows us to find a slot for a specific key index. For example if there are 10 keys in the slot 0, then we can quickly find a slot that contains 11th key using binary search on top of the binary index tree.
      * scan API - in order to perform a scan across the entire DB, the cursor now needs to not only save position within the dictionary but also the slot id. In this change we append slot id into LSB of the cursor so it can be passed around between client and the server. This has interesting side effect, now you'll be able to start scanning specific slot by simply providing slot id as a cursor value. The plan is to not document this as defined behavior, however. It's also worth nothing the SCAN API is now technically incompatible with previous versions, although practically we don't believe it's an issue.
      * Checksum calculation optimizations - During command execution, we know that all of the keys are from the same slot (outside of a few notable exceptions such as cross slot scripts and modules). We don't want to compute the checksum multiple multiple times, hence we are relying on cached slot id in the client during the command executions. All operations that access random keys, either should pass in the known slot or recompute the slot. 
      * Slot info in RDB - in order to resize individual dictionaries correctly, while loading RDB, it's not enough to know total number of keys (of course we could approximate number of keys per slot, but it won't be precise). To address this issue, we've added additional metadata into RDB that contains number of keys in each slot, which can be used as a hint during loading.
      * DB size - besides `DBSIZE` API, we need to know size of the DB in many places want, in order to avoid scanning all dictionaries and summing up their sizes in a loop, we've introduced a new field into `redisDb` that keeps track of `key_count`. This way we can keep DBSIZE operation O(1). This is also kept for O(1) expires computation as well.
      
      ## Performance
      This change improves SET performance in cluster mode by ~5%, most of the gains come from us not having to maintain linked lists for keys in slot, non-cluster mode has same performance. For workloads that rely on evictions, the performance is similar because of the extra overhead for finding keys to evict. 
      
      RDB loading performance is slightly reduced, as the slot of each key needs to be computed during the load.
      
      ## Interface changes
      * Removed `overhead.hashtable.slot-to-keys` to `MEMORY STATS`
      * Scan API will now require 64 bits to store the cursor, even on 32 bit systems, as the slot information will be stored.
      * New RDB version to support the new op code for SLOT information. 
      
      ---------
      Co-authored-by: default avatarVitaly Arbuzov <arvit@amazon.com>
      Co-authored-by: default avatarHarkrishn Patro <harkrisp@amazon.com>
      Co-authored-by: default avatarRoshan Khatri <rvkhatri@amazon.com>
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0270abda
  22. 13 Oct, 2023 2 commits
  23. 12 Oct, 2023 3 commits
    • Ye Lin Aung's avatar
      Replace `emptyDb()` with new `emptyData()` (#12646) · b705049a
      Ye Lin Aung authored
      The function was renamed, but the comments were outdated.
      b705049a
    • zhaozhao.zz's avatar
      support XREAD[GROUP] with BLOCK option in scripts (#12596) · 77a65e82
      zhaozhao.zz authored
      In #11568 we removed the NOSCRIPT flag from commands and keep the BLOCKING flag.
      Aiming to allow them in scripts and let them implicitly behave in the non-blocking way.
      
      In that sense, the old behavior was to allow LPOP and reject BLPOP, and the new behavior,
      is to allow BLPOP too, and fail it only in case it ends up blocking.
      So likewise, so far we allowed XREAD and rejected XREAD BLOCK, and we will now allow
      that too, and only reject it if it ends up blocking.
      77a65e82
    • Binbin's avatar
      Fix crash when running rebalance command in a mixed cluster of 7.0 and 7.2 (#12604) · e5ef1613
      Binbin authored
      In #10536, we introduced the assert, some older versions of servers
      (like 7.0) doesn't gossip shard_id, so we will not add the node to
      cluster->shards, and node->shard_id is filled in randomly and may not
      be found here.
      
      It causes that if we add a 7.2 node to a 7.0 cluster and allocate slots
      to the 7.2 node, the 7.2 node will crash when it hits this assert. Somehow
      like #12538.
      
      In this PR, we remove the assert and replace it with an unconditional removal.
      e5ef1613
  24. 11 Oct, 2023 1 commit
    • Binbin's avatar
      Fix redis-cli pubsub_mode and connect minor prompt / crash issue (#12571) · 4de4fcf2
      Binbin authored
      When entering pubsub mode and using the redis-cli only
      connect command, we need to reset pubsub_mode because
      we switch to a different connection.
      
      This will affect the prompt when the connection is successful,
      and redis-cli will crash when the connect fails:
      ```
      127.0.0.1:6379> subscribe ch
      1) "subscribe"
      2) "ch"
      3) (integer) 1
      127.0.0.1:6379(subscribed mode)> connect 127.0.0.1 6380
      127.0.0.1:6380(subscribed mode)> ping
      PONG
      127.0.0.1:6380(subscribed mode)> connect a b
      Could not connect to Redis at a:0: Name or service not known
      Segmentation fault
      ```
      4de4fcf2
  25. 10 Oct, 2023 1 commit
  26. 08 Oct, 2023 2 commits
    • Oran Agra's avatar
      dump server longs on hang corrupt dump fuzzer test · b810384c
      Oran Agra authored
      recently there are some incidents of hanged tests in the CI
      when we try to reproduce them, we get an assertion, not a hang.
      maybe the server logs will reveal some info.
      b810384c
    • Jachin's avatar
      Fix compile on macOS 13 (#12611) · a2b0701d
      Jachin authored
      Use the __MAC_OS_X_VERSION_MIN_REQUIRED macro to detect the
      macOS system version instead of using MAC_OS_X_VERSION_10_6.
      
      From MacOSX14.0.sdk, the default definitions of MAC_OS_X_VERSION_xxx have
      been removed in usr/include/AvailabilityMacros.h. It includes AvailabilityVersions.h,
      where the following condition must be met:
      `#if (!defined(_POSIX_C_SOURCE) && !defined(_XOPEN_SOURCE)) || defined(_DARWIN_C_SOURCE)`
      Only then will MAC_OS_X_VERSION_xxx be defined.
      However, in the project, _DARWIN_C_SOURCE is not defined, which leads to the
      loss of the definition for MAC_OS_X_VERSION_10_6.
      a2b0701d