1. 08 Sep, 2024 1 commit
    • Ozan Tezcan's avatar
      Fix flaky replication tests (#13518) · ac03e372
      Ozan Tezcan authored
      #13495 introduced a change to reply -LOADING while flushing existing db on a replica. Some of our tests are
       sensitive to this change and do no expect -LOADING reply.
      
      Fixing a couple of tests that fail time to time.
      ac03e372
  2. 03 Sep, 2024 1 commit
    • Ozan Tezcan's avatar
      Reply LOADING on replica while flushing the db (#13495) · a7afd1d2
      Ozan Tezcan authored
      On a full sync, replica starts discarding existing db. If the existing 
      db is huge and flush is happening synchronously, replica may become 
      unresponsive. 
      
      Adding a change to yield back to event loop while flushing db on 
      a replica. Replica will reply -LOADING in this case. Note that while 
      replica is loading the new rdb, it may get an error and start flushing
      the partial db. This step may take a long time as well. Similarly, 
      replica will reply -LOADING in this case. 
      
      To call processEventsWhileBlocked() and reply -LOADING, we need to do:
      - Set connSetReadHandler() null not to process further data from the master
      - Set server.loading flag
      - Call blockingOperationStarts()
      
      rdbload() already does these steps and calls processEventsWhileBlocked()
      while loading the rdb. Added a new call rdbLoadWithEmptyFunc() which 
      accepts callback to flush db before loading rdb or when an error 
      happens while loading. 
      
      For diskless replication, doing something similar and calling emptyData()
      after setting required flags.
      
      Additional changes:
      - Allow `appendonly` config change during loading. 
       Config can be changed while loading data on startup or on replication 
       when slave is loading RDB. We allow config change command to update 
       `server.aof_enabled` and then lazily apply config change after loading
       operation is completed.
       
       - Added a test for `replica-lazy-flush` config
      a7afd1d2
  3. 01 Aug, 2024 1 commit
  4. 04 Jun, 2024 1 commit
    • gms's avatar
      Fix crash due to unblock client during slot migration (#13311) · f36b5a85
      gms authored
      
      
      In #13224, we found a crash during cluster slot migration but don't know
      why. So i check all the return C_OK in processCommand to see if we are
      missing some duration reset and see this.
      
      This fix is like #12247, when we reject the command, we should reset the
      duration. I test it and verify it can fix #13224.
      
      So the reason may because we are using stream block and then during the
      slot migration, it got a redirect and then crash the server.
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      f36b5a85
  5. 20 Mar, 2024 1 commit
  6. 27 Dec, 2023 1 commit
    • Chen Tianjie's avatar
      Replace slots_to_channels radix tree with slot specific dictionaries for shard channels. (#12804) · 85279595
      Chen Tianjie authored
      
      
      We have achieved replacing `slots_to_keys` radix tree with key->slot
      linked list (#9356), and then replacing the list with slot specific
      dictionaries for keys (#11695).
      
      Shard channels behave just like keys in many ways, and we also need a
      slots->channels mapping. Currently this is still done by using a radix
      tree. So we should split `server.pubsubshard_channels` into 16384 dicts
      and drop the radix tree, just like what we did to DBs.
      
      Some benefits (basically the benefits of what we've done to DBs):
      1. Optimize counting channels in a slot. This is currently used only in
      removing channels in a slot. But this is potentially more useful:
      sometimes we need to know how many channels there are in a specific slot
      when doing slot migration. Counting is now implemented by traversing the
      radix tree, and with this PR it will be as simple as calling `dictSize`,
      from O(n) to O(1).
      2. The radix tree in the cluster has been removed. The shard channel
      names no longer require additional storage, which can save memory.
      3. Potentially useful in slot migration, as shard channels are logically
      split by slots, thus making it easier to migrate, remove or add as a
      whole.
      4. Avoid rehashing a big dict when there is a large number of channels.
      
      Drawbacks:
      1. Takes more memory than using radix tree when there are relatively few
      shard channels.
      
      What this PR does:
      1. in cluster mode, split `server.pubsubshard_channels` into 16384
      dicts, in standalone mode, still use only one dict.
      2. drop the `slots_to_channels` radix tree.
      3. to save memory (to solve the drawback above), all 16384 dicts are
      created lazily, which means only when a channel is about to be inserted
      to the dict will the dict be initialized, and when all channels are
      deleted, the dict would delete itself.
      5. use `server.shard_channel_count` to keep track of the number of all
      shard channels.
      
      ---------
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      85279595
  7. 19 Nov, 2023 1 commit
  8. 13 Oct, 2023 1 commit
  9. 06 Jul, 2023 1 commit
    • Sankar's avatar
      Process loss of slot ownership in cluster bus (#12344) · 1190f25c
      Sankar authored
      Process loss of slot ownership in cluster bus
      
      When a node no longer owns a slot, it clears the bit corresponding
      to the slot in the cluster bus messages. The receiving nodes
      currently don't record the fact that the sender stopped claiming
      a slot until some other node in the cluster starts claiming the slot.
      This can cause a slot to go missing during slot migration when subjected
      to inopportune race with addition of new shards or a failover.
      This fix forces the receiving nodes to process the loss of ownership
      to avoid spreading wrong information.
      1190f25c
  10. 13 Jun, 2023 1 commit
  11. 24 May, 2023 1 commit
    • Binbin's avatar
      Add dummy CLUSTER SLAVES call tests to fix reply ci (#12224) · ec5721d6
      Binbin authored
      In #12166, we removed a call to CLUSTER SLAVES, which
      then caused reply-schemas ci to fail:
      ```
      WARNING! The following commands were not hit at all:
        cluster|slaves
        ERROR! at least one command was not hit by the tests
      ```
      
      Because we already have command output that cover CLUSTER REPLICAS
      elsewhere, here we simply add some dummy tests to fix the ci.
      ec5721d6
  12. 23 May, 2023 1 commit
  13. 18 Apr, 2023 1 commit
  14. 11 Mar, 2023 1 commit
    • guybe7's avatar
      Add reply_schema to command json files (internal for now) (#10273) · 4ba47d2d
      guybe7 authored
      Work in progress towards implementing a reply schema as part of COMMAND DOCS, see #9845
      Since ironing the details of the reply schema of each and every command can take a long time, we
      would like to merge this PR when the infrastructure is ready, and let this mature in the unstable branch.
      Meanwhile the changes of this PR are internal, they are part of the repo, but do not affect the produced build.
      
      ### Background
      In #9656 we add a lot of information about Redis commands, but we are missing information about the replies
      
      ### Motivation
      1. Documentation. This is the primary goal.
      2. It should be possible, based on the output of COMMAND, to be able to generate client code in typed
        languages. In order to do that, we need Redis to tell us, in detail, what each reply looks like.
      3. We would like to build a fuzzer that verifies the reply structure (for now we use the existing
        testsuite, see the "Testing" section)
      
      ### Schema
      The idea is to supply some sort of schema for the various replies of each command.
      The schema will describe the conceptual structure of the reply (for generated clients), as defined in RESP3.
      Note that the reply structure itself may change, depending on the arguments (e.g. `XINFO STREAM`, with
      and without the `FULL` modifier)
      We decided to use the standard json-schema (see https://json-schema.org/) as the reply-schema.
      
      Example for `BZPOPMIN`:
      ```
      "reply_schema": {
          "oneOf": [
              {
                  "description": "Timeout reached and no elements were popped.",
                  "type": "null"
              },
              {
                  "description": "The keyname, popped member, and its score.",
                  "type": "array",
                  "minItems": 3,
                  "maxItems": 3,
                  "items": [
                      {
                          "description": "Keyname",
                          "type": "string"
                      },
                      {
                          "description": "Member",
                          "type": "string"
                      },
                      {
                          "description": "Score",
                          "type": "number"
                      }
                  ]
              }
          ]
      }
      ```
      
      #### Notes
      1.  It is ok that some commands' reply structure depends on the arguments and it's the caller's responsibility
        to know which is the relevant one. this comes after looking at other request-reply systems like OpenAPI,
        where the reply schema can also be oneOf and the caller is responsible to know which schema is the relevant one.
      2. The reply schemas will describe RESP3 replies only. even though RESP3 is structured, we want to use reply
        schema for documentation (and possibly to create a fuzzer that validates the replies)
      3. For documentation, the description field will include an explanation of the scenario in which the reply is sent,
        including any relation to arguments. for example, for `ZRANGE`'s two schemas we will need to state that one
        is with `WITHSCORES` and the other is without.
      4. For documentation, there will be another optional field "notes" in which we will add a short description of
        the representation in RESP2, in case it's not trivial (RESP3's `ZRANGE`'s nested array vs. RESP2's flat
        array, for example)
      
      Given the above:
      1. We can generate the "return" section of all commands in [redis-doc](https://redis.io/commands/)
        (given that "description" and "notes" are comprehensive enough)
      2. We can generate a client in a strongly typed language (but the return type could be a conceptual
        `union` and the caller needs to know which schema is relevant). see the section below for RESP2 support.
      3. We can create a fuzzer for RESP3.
      
      ### Limitations (because we are using the standard json-schema)
      The problem is that Redis' replies are more diverse than what the json format allows. This means that,
      when we convert the reply to a json (in order to validate the schema against it), we lose information (see
      the "Testing" section below).
      The other option would have been to extend the standard json-schema (and json format) to include stuff
      like sets, bulk-strings, error-string, etc. but that would mean also extending the schema-validator - and that
      seemed like too much work, so we decided to compromise.
      
      Examples:
      1. We cannot tell the difference between an "array" and a "set"
      2. We cannot tell the difference between simple-string and bulk-string
      3. we cannot verify true uniqueness of items in commands like ZRANGE: json-schema doesn't cover the
        case of two identical members with different scores (e.g. `[["m1",6],["m1",7]]`) because `uniqueItems`
        compares (member,score) tuples and not just the member name. 
      
      ### Testing
      This commit includes some changes inside Redis in order to verify the schemas (existing and future ones)
      are indeed correct (i.e. describe the actual response of Redis).
      To do that, we added a debugging feature to Redis that causes it to produce a log of all the commands
      it executed and their replies.
      For that, Redis needs to be compiled with `-DLOG_REQ_RES` and run with
      `--reg-res-logfile <file> --client-default-resp 3` (the testsuite already does that if you run it with
      `--log-req-res --force-resp3`)
      You should run the testsuite with the above args (and `--dont-clean`) in order to make Redis generate
      `.reqres` files (same dir as the `stdout` files) which contain request-response pairs.
      These files are later on processed by `./utils/req-res-log-validator.py` which does:
      1. Goes over req-res files, generated by redis-servers, spawned by the testsuite (see logreqres.c)
      2. For each request-response pair, it validates the response against the request's reply_schema
        (obtained from the extended COMMAND DOCS)
      5. In order to get good coverage of the Redis commands, and all their different replies, we chose to use
        the existing redis test suite, rather than attempt to write a fuzzer.
      
      #### Notes about RESP2
      1. We will not be able to use the testing tool to verify RESP2 replies (we are ok with that, it's time to
        accept RESP3 as the future RESP)
      2. Since the majority of the test suite is using RESP2, and we want the server to reply with RESP3
        so that we can validate it, we will need to know how to convert the actual reply to the one expected.
         - number and boolean are always strings in RESP2 so the conversion is easy
         - objects (maps) are always a flat array in RESP2
         - others (nested array in RESP3's `ZRANGE` and others) will need some special per-command
           handling (so the client will not be totally auto-generated)
      
      Example for ZRANGE:
      ```
      "reply_schema": {
          "anyOf": [
              {
                  "description": "A list of member elements",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "string"
                  }
              },
              {
                  "description": "Members and their scores. Returned in case `WITHSCORES` was used.",
                  "notes": "In RESP2 this is returned as a flat array",
                  "type": "array",
                  "uniqueItems": true,
                  "items": {
                      "type": "array",
                      "minItems": 2,
                      "maxItems": 2,
                      "items": [
                          {
                              "description": "Member",
                              "type": "string"
                          },
                          {
                              "description": "Score",
                              "type": "number"
                          }
                      ]
                  }
              }
          ]
      }
      ```
      
      ### Other changes
      1. Some tests that behave differently depending on the RESP are now being tested for both RESP,
        regardless of the special log-req-res mode ("Pub/Sub PING" for example)
      2. Update the history field of CLIENT LIST
      3. Added basic tests for commands that were not covered at all by the testsuite
      
      ### TODO
      
      - [x] (maybe a different PR) add a "condition" field to anyOf/oneOf schemas that refers to args. e.g.
        when `SET` return NULL, the condition is `arguments.get||arguments.condition`, for `OK` the condition
        is `!arguments.get`, and for `string` the condition is `arguments.get` - https://github.com/redis/redis/issues/11896
      - [x] (maybe a different PR) also run `runtest-cluster` in the req-res logging mode
      - [x] add the new tests to GH actions (i.e. compile with `-DLOG_REQ_RES`, run the tests, and run the validator)
      - [x] (maybe a different PR) figure out a way to warn about (sub)schemas that are uncovered by the output
        of the tests - https://github.com/redis/redis/issues/11897
      - [x] (probably a separate PR) add all missing schemas
      - [x] check why "SDOWN is triggered by misconfigured instance replying with errors" fails with --log-req-res
      - [x] move the response transformers to their own file (run both regular, cluster, and sentinel tests - need to
        fight with the tcl including mechanism a bit)
      - [x] issue: module API - https://github.com/redis/redis/issues/11898
      - [x] (probably a separate PR): improve schemas: add `required` to `object`s - https://github.com/redis/redis/issues/11899
      
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarHanna Fadida <hanna.fadida@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarShaya Potter <shaya@redislabs.com>
      4ba47d2d
  15. 23 Dec, 2022 1 commit
    • Madelyn Olson's avatar
      Harden init-tests for cluster tests (#11635) · 7379d221
      Madelyn Olson authored
      Attempt to harden cluster init-tests by doing two things:
      * Retry up to 3 times to join the cluster. Cluster meet is entirely idempotent, so it should stabilize if we missed a node.
      * Validate the connection is actually established, not just exists in the cluster list. Nodes can exist in handshake, but might later get dropped.
      7379d221
  16. 17 Nov, 2022 1 commit
    • Ping Xie's avatar
      Introduce Shard IDs to logically group nodes in cluster mode (#10536) · 203b12e4
      Ping Xie authored
      Introduce Shard IDs to logically group nodes in cluster mode.
      1. Added a new "shard_id" field to "cluster nodes" output and nodes.conf after "hostname"
      2. Added a new PING extension to propagate "shard_id"
      3. Handled upgrade from pre-7.2 releases automatically
      4. Refactored PING extension assembling/parsing logic
      
      Behavior of Shard IDs:
      
      Replicas will always follow the shards of their reported primaries. If a primary updates its shard ID, the replica will follow. (This need not follow for cluster v2) This is not an expected use case.
      203b12e4
  17. 02 Nov, 2022 1 commit
  18. 05 Sep, 2022 1 commit
  19. 12 Jul, 2022 1 commit
  20. 11 Jul, 2022 2 commits
  21. 04 Jul, 2022 1 commit
  22. 29 Jun, 2022 1 commit
  23. 31 May, 2022 1 commit
    • Harkrishn Patro's avatar
      Sharded pubsub publish messagebulk as smessage (#10792) · 4065b4f2
      Harkrishn Patro authored
      To easily distinguish between sharded channel message and a global
      channel message, introducing `smessage` (instead of `message`) as
      message bulk for sharded channel publish message.
      
      This is gonna be a breaking change in 7.0.1!
      
      Background:
      Sharded pubsub introduced in redis 7.0, but after the release we quickly
      realized that the fact that it's problematic that the client can't distinguish
      between normal (global) pubsub messages and sharded ones.
      This is important because the same connection can subscribe to both,
      but messages sent to one pubsub system are not propagated to the
      other (they're completely separate), so if one connection is used to
      subscribe to both, we need to assist the client library to know which
      message it got so it can forward it to the correct callback.
      4065b4f2
  24. 05 Apr, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Functions: Move library meta data to be part of the library payload. (#10500) · ae020e3d
      Meir Shpilraien (Spielrein) authored
      ## Move library meta data to be part of the library payload.
      
      Following the discussion on https://github.com/redis/redis/issues/10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.**
      
      The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example:
      
      ```lua
      #!lua name=test
      redis.register_function('foo', function() return 1 end)
      ```
      
      The above script will run on the Lua engine and will create a library called `test`.
      
      ## API Changes (compare to 7.0 rc2)
      
      * `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`.
      * The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST`
      
      
      ## Breaking Changes (compare to 7.0 rc2)
      
      * Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line).
      * Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid.
      
      ## Notes
      
      * Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).
      ae020e3d
  25. 02 Apr, 2022 1 commit
    • Viktor Söderqvist's avatar
      Turn into replica on SETSLOT (#10489) · b53c7f2c
      Viktor Söderqvist authored
      * Fix race condition where node loses its last slot and turns into replica
      
      When a node has lost its last slot and finds out from the SETSLOT command
      before the cluster bus PONG from the new owner arrives. In this case, the
      node didn't turn itself into a replica of the new slot owner.
      
      This commit adds the same logic to the SETSLOT command as already exists
      for the cluster bus PONG processing.
      
      * Revert "Fix new / failing cluster slot migration test (#10482)"
      
      This reverts commit 0b21ef8d.
      
      In this test, the old slot owner finds out that it has lost its last
      slot in a nondeterministic way. Either the cluster bus PONG from the
      new slot owner and sometimes in a SETSLOT command from redis-cli. In
      both cases, the result should be the same and the old owner should
      turn itself into a replica of the new slot owner.
      b53c7f2c
  26. 31 Mar, 2022 1 commit
    • Madelyn Olson's avatar
      Prevent replica failover during manual takeover test (#10499) · e81bd15e
      Madelyn Olson authored
      During 11-manual-takeover.tcl, if the killing of the instances happens
      too slowly, one of the replicas might be able to promote itself.
      I'm not sure why it was slow, but it was observed taking 6 seconds
      which is enough time to do an election.
      I was able to verify the error locally by adding a small delay (1 second)
      during ASAN CI. A fix is just to disable automated failover until all the
      nodes are confirmed dead.
      e81bd15e
  27. 21 Mar, 2022 1 commit
  28. 16 Mar, 2022 2 commits
  29. 11 Feb, 2022 1 commit
  30. 04 Feb, 2022 1 commit
  31. 03 Feb, 2022 1 commit
  32. 31 Jan, 2022 1 commit
  33. 30 Jan, 2022 2 commits
    • Oran Agra's avatar
      Revent the attempt to fix cluster rebalance test (#10207) (#10212) · d364ede5
      Oran Agra authored
      It seems that fix didn't really solve the problem with ASAN,
      and also introduced issues with other CI runs.
      
      unrelated:
      - make runtest-cluster able to take multiple --single arguments
      d364ede5
    • Oran Agra's avatar
      fix cluster rebalance test race (#10207) · be0d2933
      Oran Agra authored
      Try to fix the rebalance cluster test that's failing with ASAN daily:
      
      Looks like `redis-cli --cluster rebalance` gets `ERR Please use SETSLOT only with masters` in `clusterManagerMoveSlot()`.
      it happens when `12-replica-migration-2.tcl` is run with ASAN in GH Actions.
      in `Resharding all the master #0 slots away from it`
      
      So the fix (assuming i got it right) is to call `redis-cli --cluster check` before `--cluster rebalance`.
      p.s. it looks like a few other checks in these tests needed that wait, added them too.
      
      Other changes:
      * in instances.tcl, make sure to catch tcl test crashes and let the rest of the code proceed, so that if there was
        a redis crash, we'll find it and print it too.
      * redis-cli, try to make sure it prints an error instead of silently exiting.
      
      specifically about redis-cli:
      1. clusterManagerMoveSlot used to print an error, only if the caller also asked for it (should be the other way around).
      2. clusterManagerCommandReshard asked for an error, but didn't use it (probably tried to avoid the double print).
      3. clusterManagerCommandRebalance didn't ask for the error, now it does.
      4. making sure that other places in clusterManagerCommandRebalance print something before exiting with an error.
      be0d2933
  34. 26 Jan, 2022 1 commit
    • Madelyn Olson's avatar
      Change expression to look for at least one limit exceeded (#10173) · f6b76e50
      Madelyn Olson authored
      This is an attempt to fix some of the issues with the cluster mode tests we are seeing in the daily run.
      
      The test is trying to incrementally adds a bunch of publish messages, expecting that eventually one
      of them will overflow. The tests stops one of the processes, so it expects that just that one Redis node
      will overflow. I think the test is flaky because under certain circumstances multiple links are getting
      disconnected, not just the one that is stalled.
      f6b76e50
  35. 24 Jan, 2022 2 commits
    • yoav-steinberg's avatar
      Support function flags in script EVAL via shebang header (#10126) · 7eadc5ee
      yoav-steinberg authored
      In #10025 we added a mechanism for flagging certain properties for Redis Functions.
      This lead us to think we'd like to "port" this mechanism to Redis Scripts (`EVAL`) as well. 
      
      One good reason for this, other than the added functionality is because it addresses the
      poor behavior we currently have in `EVAL` in case the script performs a (non DENY_OOM) write operation
      during OOM state. See #8478 (And a previous attempt to handle it via #10093) for details.
      Note that in Redis Functions **all** write operations (including DEL) will return an error during OOM state
      unless the function is flagged as `allow-oom` in which case no OOM checking is performed at all.
      
      This PR:
      - Enables setting `EVAL` (and `SCRIPT LOAD`) script flags as defined in #10025.
      - Provides a syntactical framework via [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) for
        additional script annotations and even engine selection (instead of just lua) for scripts.
      - Provides backwards compatibility so scripts without the new annotations will behave as they did before.
      - Appropriate tests.
      - Changes `EVAL[SHA]/_RO` to be flagged as `STALE` commands. This makes it possible to flag individual
        scripts as `allow-stale` or not flag them as such. In backwards compatibility mode these commands will
        return the `MASTERDOWN` error as before.
      - Changes `SCRIPT LOAD` to be flagged as a `STALE` command. This is mainly to make it logically
        compatible with the change to `EVAL` in the previous point. It enables loading a script on a stale server
        which is technically okay it doesn't relate directly to the server's dataset. Running the script does, but that
        won't work unless the script is explicitly marked as `allow-stale`.
      
      Note that even though the LUA syntax doesn't support hash tag comments `.lua` files do support a shebang
      tag on the top so they can be executed on Unix systems like any shell script. LUA's `luaL_loadfile` handles
      this as part of the LUA library. In the case of `luaL_loadbuffer`, which is what Redis uses, I needed to fix the
      input script in case of a shebang manually. I did this the same way `luaL_loadfile` does, by replacing the
      first line with a single line feed character.
      7eadc5ee
    • ny0312's avatar
      Fix flaky cluster tests in 24-links.tcl (#10157) · b40a9ba5
      ny0312 authored
      
      
      * Fix flaky cluster test "Disconnect link when send buffer limit reached"
      * Fix flaky cluster test "Each node has two links with each peer"
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      b40a9ba5
  36. 19 Jan, 2022 1 commit