1. 11 Jan, 2024 1 commit
  2. 10 Jan, 2024 1 commit
  3. 09 Jan, 2024 2 commits
  4. 08 Jan, 2024 5 commits
    • Binbin's avatar
      Fix minor fd leak in rdbSaveToSlavesSockets (#12919) · 14e4a983
      Binbin authored
      We should close server.rdb_child_exit_pipe when redisFork fails,
      otherwise the pipe fd will be leaked.
      
      Just a cleanup.
      14e4a983
    • Andy Pan's avatar
      Re-indent code and reduce code being complied on Solaris for anetKeepAlive (#12914) · 50b8b997
      Andy Pan authored
      This is a follow-up PR for #12782, in which we introduced nested
      preprocessor directives for TCP keep-alive on Solaris and added
      redundant indentation for code. Besides, it could result in unreachable
      code due to the lack of `#else` on the latest Solaris 11.4 where
      `TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` are available. As a
      result, this PR does three main things:
      
      - To eliminate the redundant indention for C code in nested preprocessor
      directives
      - To add `#else` directives and move `TCP_KEEPALIVE_THRESHOLD` +
      `TCP_KEEPALIVE_ABORT_THRESHOLD` settings under it, avoid unreachable
      code and compiler warnings when `#if defined(TCP_KEEPIDLE) &&
      defined(TCP_KEEPINTVL) && defined(TCP_KEEPCNT)` is met on Solaris 11.4
      - To remove a few trailing whitespace in comments
      50b8b997
    • Yanqi Lv's avatar
      Optimize performance when many clients [p|s]unsubscribe simultaneously (#12838) · c452e414
      Yanqi Lv authored
      I'm testing the performance of Pub/Sub command recently. I find if many
      clients unsubscribe or are killed simultaneously, Redis needs a long
      time to deal with it.
      
      In my experiment, I set 5000 clients and each client subscribes 100
      channels. Then I call `client kill type pubsub` to simulate the
      situation where clients unsubscribe all channels at the same time and
      calculate the execution time. The result shows that it takes about 23s.
      I use the _perf_ and find that `listSearchKey` in
      `pubsubUnsubscribeChannel` costs more than 90% cpu time. I think we can
      optimize this situation.
      
      In this PR, I replace list with dict to track the clients subscribing
      the channel more efficiently. It changes O(N) to O(1) in the search
      phase. Then I repeat the experiment as above. The results are as
      follows.
      
      |              | Execution Time(s) |used_memory(MB) |
      | :---------------- | :------: | :----: |
      | unstable(1bd0b549)        |   23.734   | 65.41 |
      | optimize-pubsub           |   0.288   | 67.66 |
      
      Thanks for #11595 , I use a no-value dict and the results shows that the
      performance improves significantly but the memory usage only increases
      slightly.
      
      Notice:
      
      - This PR will cause the performance degradation about 20% in
      `[p|s]subscribe` command but won't freeze Redis.
      c452e414
    • debing.sun's avatar
      Change destination key's key-spec flag from RW to OW for SINTERSTORE command (#12917) · 4730563e
      debing.sun authored
      In #10122, we set the destination key's flag of SINTERSTORE to `RW`, 
      however, this command doesn't actually read or modify the destination
      key, just overwrites it.
      Therefore, we change it to `OW` similarly to all other *STORE commands.
      4730563e
    • Binbin's avatar
      Fix CLUSTER SHARDS crash in 7.0/7.2 mixed clusters where shard ids are not sync (#12832) · 5b0c6a82
      Binbin authored
      Crash reported in #12695. In the process of upgrading the cluster from
      7.0 to 7.2, because the 7.0 nodes will not gossip shard id, in 7.2 we
      will rely on shard id to build the server.cluster->shards dict.
      
      In some cases, for example, the 7.0 master node and the 7.2 replica node.
      From the view of 7.2 replica node, the cluster->shards dictionary does not
      have its master node. In this case calling CLUSTER SHARDS on the 7.2 replica
      node may crash.
      
      We should fix the underlying assumption of updateShardId, which is that the
      shard dict should be always in sync with the node's shard_id. The fix was
      suggested by PingXie, see more details in #12695.
      5b0c6a82
  5. 07 Jan, 2024 2 commits
    • debing.sun's avatar
      Make RM_Yield thread-safe (#12905) · ca1f67af
      debing.sun authored
      ## Issues and solutions from #12817
      1. Touch ProcessingEventsWhileBlocked and calling moduleCount() without
      GIL in afterSleep()
          - Introduced: 
             Version: 7.0.0
             PR: #9963
      
         - Harm Level: Very High
      If the module thread calls `RM_Yield()` before the main thread enters
      afterSleep(),
      and modifies `ProcessingEventsWhileBlocked`(+1), it will cause the main
      thread to not wait for GIL,
      which can lead to all kinds of unforeseen problems, including memory
      data corruption.
      
         - Initial / Abandoned Solution:
            * Added `__thread` specifier for ProcessingEventsWhileBlocked.
      `ProcessingEventsWhileBlocked` is used to protect against nested event
      processing, but event processing
      in the main thread and module threads should be completely independent
      and unaffected, so it is safer
               to use TLS.
      * Adding a cached module count to keep track of the current number of
      modules, to avoid having to use `dictSize()`.
          
          - Related Warnings:
      ```
      WARNING: ThreadSanitizer: data race (pid=1136)
        Write of size 4 at 0x0001045990c0 by thread T4 (mutexes: write M0):
          #0 processEventsWhileBlocked networking.c:4135 (redis-server:arm64+0x10006d124)
          #1 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
          #2 bg_call_worker <null>:83232836 (blockedclient.so:arm64+0x16a8)
      
        Previous read of size 4 at 0x0001045990c0 by main thread:
          #0 afterSleep server.c:1861 (redis-server:arm64+0x100024f98)
          #1 aeProcessEvents ae.c:408 (redis-server:arm64+0x10000fd64)
          #2 aeMain ae.c:496 (redis-server:arm64+0x100010f0c)
          #3 main server.c:7220 (redis-server:arm64+0x10003f38c)
      ```
      
      2. aeApiPoll() is not thread-safe
      When using RM_Yield to handle events in a module thread, if the main
      thread has not yet
      entered `afterSleep()`, both the module thread and the main thread may
      touch `server.el` at the same time.
      
          - Introduced: 
             Version: 7.0.0
             PR: #9963
      
         - Old / Abandoned Solution:
      Adding a new mutex to protect timing between after beforeSleep() and
      before afterSleep().
      Defect: If the main thread enters the ae loop without any IO events, it
      will wait until
      the next timeout or until there is any event again, and the module
      thread will
      always hang until the main thread leaves the event loop.
      
          - Related Warnings:
      ```
      SUMMARY: ThreadSanitizer: data race ae_kqueue.c:55 in addEventMask
      ==================
      ==================
      WARNING: ThreadSanitizer: data race (pid=14682)
        Write of size 4 at 0x000100b54000 by thread T9 (mutexes: write M0):
          #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
          #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
          #2 processEventsWhileBlocked networking.c:4138 (redis-server:arm64+0x10006d3c4)
          #3 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
          #4 bg_call_worker <null>:16042052 (blockedclient.so:arm64+0x169c)
      
        Previous write of size 4 at 0x000100b54000 by main thread:
          #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
          #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
          #2 aeMain ae.c:496 (redis-server:arm64+0x100010da8)
          #3 main server.c:7238 (redis-server:arm64+0x10003f51c)
      ```
      
      ## The final fix as the comments:
      https://github.com/redis/redis/pull/12817#discussion_r1436427232
      Optimized solution based on the above comment:
      
      First, we add `module_gil_acquring` to indicate whether the main thread
      is currently in the acquiring GIL state.
      
      When the module thread starts to yield, there are two possibilities(we
      assume the caller keeps the GIL):
      1. The main thread is in the mid of beforeSleep() and afterSleep(), that
      is, `module_gil_acquring` is not 1 now.
      At this point, the module thread will wake up the main thread through
      the pipe and leave the yield,
      waiting for the next yield when the main thread may already in the
      acquiring GIL state.
          
      2. The main thread is in the acquiring GIL state.
      The module thread release the GIL, yielding CPU to give the main thread
      an opportunity to start
      event processing, and then acquire the GIL again until the main thread
      releases it.
      This is what
      https://github.com/redis/redis/pull/12817#discussion_r1436427232
      
      
      mentioned direction.
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ca1f67af
    • Binbin's avatar
      Use shard-id of the master if the replica does not support shard-id (#12805) · 4cae66f5
      Binbin authored
      If there are nodes in the cluster that do not support shard-id, they
      will gossip shard-id. From the perspective of nodes that support shard-id,
      their shard-id is meaningless (since shard-id is randomly generated when
      we create a node.)
      
      Nodes that support shard-id will save the shard-id information in nodes.conf.
      If the node is restarted according to nodes.conf, the server will report a
      corrupted cluster config file error. Because auxShardIdSetter will reject
      configurations with inconsistent master-replica shard-ids.
      
      A cluster-wide consensus for the node's shard_id is not necessary. The key
      is maintaining consistency of the shard_id on each individual 7.2 node.
      As the cluster progressively upgrades to version 7.2, we can expect the
      shard_ids across all nodes to naturally converge and align.
      
      In this PR, when processing the gossip, if sender is a replica and does not
      support shard-id, set the shard_id to the shard_id of its master.
      4cae66f5
  6. 04 Jan, 2024 1 commit
  7. 03 Jan, 2024 2 commits
    • Lior Kogan's avatar
      Update CONTRIBUTING.md (#12907) · 51898383
      Lior Kogan authored
      - Referring to Redis Discord channel instead of the mailing list.
      - Referring to the licensing instead of repeating it.
      51898383
    • Madelyn Olson's avatar
      Handle recursive serverAsserts and provide more information for recursive segfaults (#12857) · 068051e3
      Madelyn Olson authored
      This change is trying to make two failure modes a bit easier to deep dive:
      1. If a serverPanic or serverAssert occurs during the info (or module)
      printing, it will recursively panic, which is a lot of fun as it will
      just keep recursively printing. It will eventually stack overflow, but
      will generate a lot of text in the process.
      2. When a segfault happens during the segfault handler, no information
      is communicated other than it happened. This can be problematic because
      `info` may help diagnose the real issue, but without fixing the
      recursive crash it might be hard to get at that info.
      068051e3
  8. 02 Jan, 2024 1 commit
    • AshMosh's avatar
      Manage number of new connections per cycle (#12178) · c3f8b542
      AshMosh authored
      
      
      There are situations (especially in TLS) in which the engine gets too occupied managing a large number of new connections. Existing connections may time-out while the server is processing the new connections initial TLS handshakes, which may cause cause new connections to be established, perpetuating the problem. To better manage the tradeoff between new connection rate and other workloads, this change adds a new config to manage maximum number of new connections per event loop cycle, instead of using a predetermined number (currently 1000).
      
      This change introduces two new configurations, max-new-connections-per-cycle and max-new-tls-connections-per-cycle. The default value of the tcp connections is 10 per cycle and the default value of tls connections per cycle is 1.
      ---------
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      c3f8b542
  9. 28 Dec, 2023 5 commits
  10. 27 Dec, 2023 3 commits
    • Chen Tianjie's avatar
      Replace slots_to_channels radix tree with slot specific dictionaries for shard channels. (#12804) · 85279595
      Chen Tianjie authored
      
      
      We have achieved replacing `slots_to_keys` radix tree with key->slot
      linked list (#9356), and then replacing the list with slot specific
      dictionaries for keys (#11695).
      
      Shard channels behave just like keys in many ways, and we also need a
      slots->channels mapping. Currently this is still done by using a radix
      tree. So we should split `server.pubsubshard_channels` into 16384 dicts
      and drop the radix tree, just like what we did to DBs.
      
      Some benefits (basically the benefits of what we've done to DBs):
      1. Optimize counting channels in a slot. This is currently used only in
      removing channels in a slot. But this is potentially more useful:
      sometimes we need to know how many channels there are in a specific slot
      when doing slot migration. Counting is now implemented by traversing the
      radix tree, and with this PR it will be as simple as calling `dictSize`,
      from O(n) to O(1).
      2. The radix tree in the cluster has been removed. The shard channel
      names no longer require additional storage, which can save memory.
      3. Potentially useful in slot migration, as shard channels are logically
      split by slots, thus making it easier to migrate, remove or add as a
      whole.
      4. Avoid rehashing a big dict when there is a large number of channels.
      
      Drawbacks:
      1. Takes more memory than using radix tree when there are relatively few
      shard channels.
      
      What this PR does:
      1. in cluster mode, split `server.pubsubshard_channels` into 16384
      dicts, in standalone mode, still use only one dict.
      2. drop the `slots_to_channels` radix tree.
      3. to save memory (to solve the drawback above), all 16384 dicts are
      created lazily, which means only when a channel is about to be inserted
      to the dict will the dict be initialized, and when all channels are
      deleted, the dict would delete itself.
      5. use `server.shard_channel_count` to keep track of the number of all
      shard channels.
      
      ---------
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      85279595
    • Moshe Kaplan's avatar
      config.c: Avoid leaking file handle if file is 0 bytes (#12828) · fa751f9b
      Moshe Kaplan authored
      If fopen() is successful and redis_fstat determines that the file is 0
      bytes, the file handle stored in fp will leak. This change closes the
      filehandle stored in fp if the file is 0 bytes.
      
      Second attempt at fixing Coverity 390029
      
      This is a follow-up to #12796
      fa751f9b
    • sundb's avatar
      Fix oom-score-adj test due to no permission (#12887) · bef57153
      sundb authored
      
      
      Fix #12792
      
      On ubuntu 23(lunar), non-root users will not be allowed to change the
      oom_score_adj of a process to a value that is too low.
      Since terminal's default oom_score_adj is 200, if we run the test on
      terminal, we won't be able to set the oom_score_adj of the redis process
      to 9 or 22, which is too low.
      
      Reproduction on ubuntu 23(lunar) terminal:
      ```sh
      $ cat /proc/`pgrep redis-server`/oom_score_adj
      200
      $ echo 100 > /proc/`pgrep redis-server`/oom_score_adj
      # success without error
      $ echo 99 > /proc/`pgrep redis-server`/oom_score_adj
      echo: write error: Permission denied
      ```
      
      As from the output above, we can only set the minimum oom score of redis
      processes to 100.
      By modifying the test, make oom_score_adj only increase upwards and not
      decrease.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      bef57153
  11. 26 Dec, 2023 4 commits
  12. 24 Dec, 2023 2 commits
  13. 21 Dec, 2023 1 commit
    • Binbin's avatar
      Move cliVersion to cli_common and add --version support for redis-check-aof (#10856) · 23e980e7
      Binbin authored
      Let us see which version of redis this tool is part of.
      Similarly to redis-cli, redis-benchmark and redis-check-rdb
      
      redis-rdb-check and redis-aof-check are actually symlinks to redis,
      so they will directly use getVersion in server, the format became:
      ```
      {title} v={redis_version} sha={sha}:{dirty} malloc={malloc} bits={bits} build={build}
      ```
      
      Move cliVersion into cli_common, redis-cli and redis-benchmark will
      use it, and the format is not change:
      ```
      {title} {redis_version} (git:{sha})
      ```
      23e980e7
  14. 17 Dec, 2023 1 commit
  15. 15 Dec, 2023 2 commits
    • Binbin's avatar
      Always keep an in-memory history of all commands in redis-cli (#12862) · adbb534f
      Binbin authored
      redis-cli avoids saving sensitive commands in it's history (doesn't
      persist them to the history file).
      this means that if you had a typo and you wanna re-run the command, you
      can't easily do that.
      This PR changes that to keep an in-memory history of all the redacted
      commands, and just
      not persist them to disk. This way we would be able to press the up
      arrow and
      re-try the command freely, and it'll just not survive a redis-cli
      restart.
      adbb534f
    • zhaozhao.zz's avatar
      Unified db rehash method for both standalone and cluster (#12848) · d8a21c57
      zhaozhao.zz authored
      After #11695, we added two functions `rehashingStarted` and
      `rehashingCompleted` to the dict structure. We also registered two
      handlers for the main database's dict and expire structures. This allows
      the main database to record the dict in `rehashing` list when rehashing
      starts. Later, in `serverCron`, the `incrementallyRehash` function is
      continuously called to perform the rehashing operation. However,
      currently, when rehashing is completed, `rehashingCompleted` does not
      remove the dict from the `rehashing` list. This results in the
      `rehashing` list containing many invalid dicts. Although subsequent cron
      checks and removes dicts that don't require rehashing, it is still
      inefficient.
      
      This PR implements the functionality to remove the dict from the
      `rehashing` list in `rehashingCompleted`. This is achieved by adding
      `metadata` to the dict structure, which keeps track of its position in
      the `rehashing` list, allowing for quick removal. This approach avoids
      storing duplicate dicts in the `rehashing` list.
      
      Additionally, there are other modifications:
      
      1. Whether in standalone or cluster mode, the dict in database is
      inserted into the rehashing linked list when rehashing starts. This
      eliminates the need to distinguish between standalone and cluster mode
      in `incrementallyRehash`. The function only needs to focus on the dicts
      in the `rehashing` list that require rehashing.
      2. `rehashing` list is moved from per-database to Redis server level.
      This decouples `incrementallyRehash` from the database ID, and in
      standalone mode, there is no need to iterate over all databases,
      avoiding unnecessary access to databases that do not require rehashing.
      In the future, even if unsharded-cluster mode supports multiple
      databases, there will be no risk involved.
      3. The insertion and removal operations of dict structures in the
      `rehashing` list are decoupled from `activerehashing` config.
      `activerehashing` only controls whether `incrementallyRehash` is
      executed in serverCron. There is no need for additional steps when
      modifying the `activerehashing` switch, as in #12705.
      d8a21c57
  16. 14 Dec, 2023 1 commit
    • Guillaume Koenig's avatar
      Extend rax usage by allowing any long long value (#12837) · 967fb3c6
      Guillaume Koenig authored
      The raxFind implementation uses a special pointer value (the address of
      a static string) as the "not found" value. It works as long as actual
      pointers were used. However we've seen usages where long long,
      non-pointer values have been used. It creates a risk that one of the
      long long value precisely is the address of the special "not found"
      value. This commit changes raxFind to return 1 or 0 to indicate
      elementhood, and take in a new void **value to optionally return the
      associated value.
      
      By extension, this also allow the RedisModule_DictSet/Replace operations
      to also safely insert integers instead of just pointers.
      967fb3c6
  17. 13 Dec, 2023 3 commits
    • Chen Tianjie's avatar
      Support by/get options for sort(_ro) in cluster mode when pattern implies slot. (#12728) · e95a5d48
      Chen Tianjie authored
      The by/get options of sort/sort_ro command used to be forbidden in
      cluster mode, since we are not sure which slot the pattern may be in.
      
      As the optimization done in #12536, patterns now can be mapped to slots,
      we should allow by/get options in cluster mode when the pattern maps to
      the same slot as the key.
      e95a5d48
    • Binbin's avatar
      Redact ACL username information and mark *-key-file-pass configs as sensitive (#12860) · 3c0fd252
      Binbin authored
      In #11489, we consider acl username to be sensitive information,
      and consider the ACL GETUSER a sensitive command and remove it
      from redis-cli historyfile.
      
      This PR redact username information in ACL GETUSER and ACL DELUSER
      from SLOWLOG, and also remove ACL DELUSER from redis-cli historyfile.
      
      This PR also mark tls-key-file-pass and tls-client-key-file-pass
      as sensitive config, will redact it from SLOWLOG and also
      remove them from redis-cli historyfile.
      3c0fd252
    • Chen Tianjie's avatar
      Add metric to INFO CLIENTS: pubsub_clients. (#12849) · f9cc25c1
      Chen Tianjie authored
      In INFO CLIENTS section, we already have blocked_clients and
      tracking_clients. We should add a new metric showing the number of
      pubsub connections, which helps performance monitoring and trouble
      shooting.
      f9cc25c1
  18. 11 Dec, 2023 1 commit
    • Binbin's avatar
      Fix delKeysInSlot server events are not executed inside an execution unit (#12745) · c85a9b78
      Binbin authored
      This is a follow-up fix to #12733. We need to apply the same changes to
      delKeysInSlot. Refer to #12733 for more details.
      
      This PR contains some other minor cleanups / improvements to the test
      suite and docs.
      It uses the postnotifications test module in a cluster mode test which
      revealed a leak in the test module (fixed).
      c85a9b78
  19. 10 Dec, 2023 2 commits
    • Binbin's avatar
      Handle missing fields in dbSwapDatabases and swapMainDbWithTempDb (#12763) · 62419c01
      Binbin authored
      The change in dbSwapDatabases seems harmless. Because in non-clustered
      mode, dbBuckets calculations are strictly accurate and in cluster mode,
      we only have one DB. Modify it for uniformity (just like resize_cursor).
      
      The change in swapMainDbWithTempDb is needed in case we swap with the
      temp db, otherwise the overhead memory usage of db can be miscalculated.
      
      In addition we will swap all fields (including rehashing list), just for
      completeness (and reduce the chance of surprises in the future).
      
      Introduced in #12697.
      62419c01
    • Binbin's avatar
      Fix rehashingStarted miscalculating bucket_count in dict initialization (#12846) · e6423b7a
      Binbin authored
      In the old dictRehashingInfo implementation, for the initialization
      scenario,
      it mistakenly directly set to_size to DICTHT_SIZE(DICT_HT_INITIAL_EXP),
      which
      is 4 in our code by default.
      
      In scenarios where dictExpand directly passes the target size as
      initialization,
      the code will calculate bucket_count incorrectly. For example, in DEBUG
      POPULATE
      or RDB load scenarios, it will cause the final bucket_count to be
      initialized to
      65536 (16384 * 4), see:
      ```
      before:
      DB 0: 10000000 keys (0 volatile) in 65536 slots HT.
      
      it should be:
      DB 0: 10000000 keys (0 volatile) in 16777216 slots HT.
      ```
      
      In PR, new ht will also be initialized before calling rehashingStarted
      in
      _dictExpand, so that the calls in dictRehashingInfo can be unified.
      
      Bug was introduced in #12697.
      e6423b7a