1. 08 Nov, 2024 1 commit
    • Yuan Wang's avatar
      Dynamic event loop binding for connection structure (#13642) · 779af3ab
      Yuan Wang authored
      
      
      The IO thread has an independent event loop, so we can no longer
      hard-code the event loop to the connection, instead, we should
      dynamically select the event loop for the connection.
      
      - configure the event loop during connection creation.
      - add a new interface to allow dynamic event loop binding.
      
      For TLS connection, we need to check for any pending data on the
      connection and handle it accordingly when changing connection cross IO
      thread and main thread. This commit doesn't handle it, @sundb will
      overall support for TLS connection later.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      779af3ab
  2. 12 Sep, 2024 1 commit
    • Steve's avatar
      Avoid cluster.nodes load corruption due to shard-id generation (#13468) · d265a614
      Steve authored
      
      
      PR #13428 doesn't fully resolve an issue where corruption errors can
      still occur on loading of cluster.nodes file - seen on upgrade where
      there were no shard_ids (from old Redis), 7.2.5 loading generated new
      random ones, and persisted them to the file before gossip/handshake
      could propagate the correct ones (or some other nodes unreachable).
      This results in a primary/replica having differing shard_id in the
      cluster.nodes and then the server cannot startup - reports corruption.
      
      This PR builds on #13428 by simply ignoring the replica's shard_id in
      cluster.nodes (if it exists), and uses the replica's primary's shard_id.
      Additional handling was necessary to cover the case where the replica
      appears before the primary in cluster.nodes, where it will first use a
      generated shard_id for the primary, and then correct after it loads the
      primary cluster.nodes entry.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      d265a614
  3. 08 Aug, 2024 1 commit
  4. 06 Aug, 2024 1 commit
  5. 05 Aug, 2024 2 commits
    • Josh Hershberg's avatar
      Make cluster shards cmd implementation generic · 6d5d7541
      Josh Hershberg authored
      
      
      This and the previous commit make the cluster
      shards command a generic implementation instead of a
      specific implementation for each cluster API implementation.
      This commit (a) adds functions to the cluster API
      and (b) modifies the cluster shards cmd implementation
      to use cluster API functions instead of directly
      accessing the legacy clustering implementation.
      Signed-off-by: default avatarJosh Hershberg <yehoshua@redis.com>
      6d5d7541
    • Josh Hershberg's avatar
      Prep to make cluster shards cmd generic · e3e631f3
      Josh Hershberg authored
      
      
      This and the next following commit makes the cluster
      shards command a generic implementation instead of a
      specific implementation for each cluster API implementation.
      This commit simply moves the cluster shards implementation
      from cluster_legacy.c to cluster.c without changing the
      implementation at all. The reason for doing so was to
      help with reviewing the changes in the diff.
      Signed-off-by: default avatarJosh Hershberg <yehoshua@redis.com>
      e3e631f3
  6. 01 Aug, 2024 1 commit
  7. 20 Mar, 2024 1 commit
  8. 05 Mar, 2024 1 commit
    • Ping Xie's avatar
      Fix PONG message processing for primary-ship tracking during failovers (#13055) · 28976a90
      Ping Xie authored
      This commit updates the processing of PONG gossip messages in the
      cluster. When a node (B) becomes a replica due to a failover, its PONG
      messages include its new primary node's (A) information and B's
      configuration epoch is aligned with A's. This allows observer nodes to
      identify changes in primary-ship, addressing issues of intermediate
      states and enhancing cluster state consistency during topology changes.
      
      Fix #13018
      28976a90
  9. 14 Feb, 2024 1 commit
    • Sankar's avatar
      Do not include gossip about receiver in cluster messages (#13046) · c1d2ac2a
      Sankar authored
      The receiver does not update any of its cluster state based on gossip
      about itself. This commit explicitly avoids sending or processing gossip
      about the receiver.
      
      Currently cluster bus gossips include 10% of nodes in the cluster with a
      minimum of 3 nodes. For up to 30 node clusters, this commit makes sure
      that 1/3 of the gossip (1 out of 3 gossips) is never discarded. This
      should help with relatively faster convergence of cluster state in
      general.
      c1d2ac2a
  10. 07 Feb, 2024 1 commit
  11. 05 Feb, 2024 1 commit
    • guybe7's avatar
      Refactor the per-slot dict-array db.c into a new kvstore data structure (#12822) · 8cd62f82
      guybe7 authored
      # Description
      Gather most of the scattered `redisDb`-related code from the per-slot
      dict PR (#11695) and turn it to a new data structure, `kvstore`. i.e.
      it's a class that represents an array of dictionaries.
      
      # Motivation
      The main motivation is code cleanliness, the idea of using an array of
      dictionaries is very well-suited to becoming a self-contained data
      structure.
      This allowed cleaning some ugly code, among others: loops that run twice
      on the main dict and expires dict, and duplicate code for allocating and
      releasing this data structure.
      
      # Notes
      1. This PR reverts the part of https://github.com/redis/redis/pull/12848
      where the `rehashing` list is global (handling rehashing `dict`s is
      under the responsibility of `kvstore`, and should not be managed by the
      server)
      2. This PR also replaces the type of `server.pubsubshard_channels` from
      `dict**` to `kvstore` (original PR:
      https://github.com/redis/redis/pull/12804). After that was done,
      server.pubsub_channels was also chosen to be a `kvstore` (with only one
      `dict`, which seems odd) just to make the code cleaner by making it the
      same type as `server.pubsubshard_channels`, see
      `pubsubtype.serverPubSubChannels`
      3. the keys and expires kvstores are currenlty configured to allocate
      the individual dicts only when the first key is added (unlike before, in
      which they allocated them in advance), but they won't release them when
      the last key is deleted.
      
      Worth mentioning that due to the recent change the reply of DEBUG
      HTSTATS changed, in case no keys were ever added to the db.
      
      before:
      ```
      127.0.0.1:6379> DEBUG htstats 9
      [Dictionary HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      [Expires HT]
      Hash table 0 stats (main hash table):
      No stats available for empty dictionaries
      ```
      
      after:
      ```
      127.0.0.1:6379> DEBUG htstats 9
      [Dictionary HT]
      [Expires HT]
      ```
      8cd62f82
  12. 23 Jan, 2024 1 commit
    • Binbin's avatar
      Add sender NULL check in clusterProcessGossipSection invalid_ids case (#12980) · 07b292af
      Binbin authored
      In the following case sender may be unknown, so we need to set up a
      NULL check for sender:
      ```
      /* If this is a MEET packet from an unknown node, we still process
       * the gossip section here since we have to trust the sender because
       * of the message type. */
      if (!sender && type == CLUSTERMSG_TYPE_MEET)
          clusterProcessGossipSection(hdr,link);
      ```
      07b292af
  13. 22 Jan, 2024 1 commit
    • Brennan's avatar
      Prevent nodes with invalid IDs from being propagated through gossip (#12921) · e12f2dec
      Brennan authored
      
      
      There have been occasional instances of memory corruption (though code bugs or bit flips) leading to invalid node information being gossiped around. To prevent this invalid information spreading, we verify the node IDs in received gossip are in an acceptable format, and disregard any gossiped nodes with invalid IDs. This PR uses the existing verifyClusterNodeId function to check the validity of the gossiped node IDs and if an invalid one is encountered, logs raw byte information to help debug the corruption.
      
      ---------
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      e12f2dec
  14. 11 Jan, 2024 2 commits
  15. 08 Jan, 2024 1 commit
    • Binbin's avatar
      Fix CLUSTER SHARDS crash in 7.0/7.2 mixed clusters where shard ids are not sync (#12832) · 5b0c6a82
      Binbin authored
      Crash reported in #12695. In the process of upgrading the cluster from
      7.0 to 7.2, because the 7.0 nodes will not gossip shard id, in 7.2 we
      will rely on shard id to build the server.cluster->shards dict.
      
      In some cases, for example, the 7.0 master node and the 7.2 replica node.
      From the view of 7.2 replica node, the cluster->shards dictionary does not
      have its master node. In this case calling CLUSTER SHARDS on the 7.2 replica
      node may crash.
      
      We should fix the underlying assumption of updateShardId, which is that the
      shard dict should be always in sync with the node's shard_id. The fix was
      suggested by PingXie, see more details in #12695.
      5b0c6a82
  16. 07 Jan, 2024 1 commit
    • Binbin's avatar
      Use shard-id of the master if the replica does not support shard-id (#12805) · 4cae66f5
      Binbin authored
      If there are nodes in the cluster that do not support shard-id, they
      will gossip shard-id. From the perspective of nodes that support shard-id,
      their shard-id is meaningless (since shard-id is randomly generated when
      we create a node.)
      
      Nodes that support shard-id will save the shard-id information in nodes.conf.
      If the node is restarted according to nodes.conf, the server will report a
      corrupted cluster config file error. Because auxShardIdSetter will reject
      configurations with inconsistent master-replica shard-ids.
      
      A cluster-wide consensus for the node's shard_id is not necessary. The key
      is maintaining consistency of the shard_id on each individual 7.2 node.
      As the cluster progressively upgrades to version 7.2, we can expect the
      shard_ids across all nodes to naturally converge and align.
      
      In this PR, when processing the gossip, if sender is a replica and does not
      support shard-id, set the shard_id to the shard_id of its master.
      4cae66f5
  17. 27 Dec, 2023 1 commit
    • Chen Tianjie's avatar
      Replace slots_to_channels radix tree with slot specific dictionaries for shard channels. (#12804) · 85279595
      Chen Tianjie authored
      
      
      We have achieved replacing `slots_to_keys` radix tree with key->slot
      linked list (#9356), and then replacing the list with slot specific
      dictionaries for keys (#11695).
      
      Shard channels behave just like keys in many ways, and we also need a
      slots->channels mapping. Currently this is still done by using a radix
      tree. So we should split `server.pubsubshard_channels` into 16384 dicts
      and drop the radix tree, just like what we did to DBs.
      
      Some benefits (basically the benefits of what we've done to DBs):
      1. Optimize counting channels in a slot. This is currently used only in
      removing channels in a slot. But this is potentially more useful:
      sometimes we need to know how many channels there are in a specific slot
      when doing slot migration. Counting is now implemented by traversing the
      radix tree, and with this PR it will be as simple as calling `dictSize`,
      from O(n) to O(1).
      2. The radix tree in the cluster has been removed. The shard channel
      names no longer require additional storage, which can save memory.
      3. Potentially useful in slot migration, as shard channels are logically
      split by slots, thus making it easier to migrate, remove or add as a
      whole.
      4. Avoid rehashing a big dict when there is a large number of channels.
      
      Drawbacks:
      1. Takes more memory than using radix tree when there are relatively few
      shard channels.
      
      What this PR does:
      1. in cluster mode, split `server.pubsubshard_channels` into 16384
      dicts, in standalone mode, still use only one dict.
      2. drop the `slots_to_channels` radix tree.
      3. to save memory (to solve the drawback above), all 16384 dicts are
      created lazily, which means only when a channel is about to be inserted
      to the dict will the dict be initialized, and when all channels are
      deleted, the dict would delete itself.
      5. use `server.shard_channel_count` to keep track of the number of all
      shard channels.
      
      ---------
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      85279595
  18. 11 Dec, 2023 1 commit
    • Binbin's avatar
      Fix delKeysInSlot server events are not executed inside an execution unit (#12745) · c85a9b78
      Binbin authored
      This is a follow-up fix to #12733. We need to apply the same changes to
      delKeysInSlot. Refer to #12733 for more details.
      
      This PR contains some other minor cleanups / improvements to the test
      suite and docs.
      It uses the postnotifications test module in a cluster mode test which
      revealed a leak in the test module (fixed).
      c85a9b78
  19. 07 Dec, 2023 1 commit
    • zhaozhao.zz's avatar
      Fix replica node cannot expand dicts when loading legacy RDB (#12839) · 8e11f84d
      zhaozhao.zz authored
      When loading RDB on cluster nodes, it is necessary to consider the
      scenario where a node is a replica.
      
      For example, during a rolling upgrade, new version instances are often
      mounted as replicas on old version instances. In this case, the full
      synchronization legacy RDB does not contain slot information, and the
      new version instance, acting as a replica, should be able to handle the
      legacy RDB correctly for `dbExpand`.
      
      Additionally, renaming `getMyClusterSlotCount` to `getMyShardSlotCount`
      would be appropriate.
      
      Introduced in #11695
      8e11f84d
  20. 03 Dec, 2023 1 commit
  21. 22 Nov, 2023 13 commits
  22. 21 Nov, 2023 2 commits
  23. 01 Nov, 2023 1 commit
  24. 31 Oct, 2023 1 commit
  25. 24 Oct, 2023 1 commit