1. 30 Jul, 2023 1 commit
  2. 12 Jun, 2023 1 commit
  3. 18 Apr, 2023 1 commit
    • sundb's avatar
      Fix some compile warnings and errors when building with gcc-12 or clang (#12035) · 42c8c618
      sundb authored
      This PR is to fix the compilation warnings and errors generated by the latest
      complier toolchain, and to add a new runner of the latest toolchain for daily CI.
      
      ## Fix various compilation warnings and errors
      
      1) jemalloc.c
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      src/jemalloc.c:1028:7: warning: suspicious concatenation of string literals in an array initialization; did you mean to separate the elements with a comma? [-Wstring-concatenation]
                          "/etc/malloc.conf",
                          ^
      src/jemalloc.c:1027:3: note: place parentheses around the string literal to silence warning
                      "\"name\" of the file referenced by the symbolic link named "
                      ^
      ```
      
      REASON:  the compiler to alert developers to potential issues with string concatenation
      that may miss a comma,
      just like #9534 which misses a comma.
      
      SOLUTION: use `()` to tell the compiler that these two line strings are continuous.
      
      2) config.h
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      In file included from quicklist.c:36:
      ./config.h:319:76: warning: attribute declaration must precede definition [-Wignored-attributes]
      char *strcat(char *restrict dest, const char *restrict src) __attribute__((deprecated("please avoid use of unsafe C functions. prefer use of redis_strlcat instead")));
      ```
      
      REASON: Enabling _FORTIFY_SOURCE will cause the compiler to use `strcpy()` with check,
      it results in a deprecated attribute declaration after including <features.h>.
      
      SOLUTION: move the deprecated attribute declaration from config.h to fmacro.h before "#include <features.h>".
      
      3) networking.c
      
      COMPILER: GCC-12
      
      WARNING: 
      ```
      networking.c: In function ‘addReplyDouble.part.0’:
      networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
        876 |         dbuf[start] = '$';
            |                     ^
      networking.c:868:14: note: at offset -5 into destination object ‘dbuf’ of size 5152
        868 |         char dbuf[MAX_LONG_DOUBLE_CHARS+32];
            |              ^
      networking.c:876:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
        876 |         dbuf[start] = '$';
            |                     ^
      networking.c:868:14: note: at offset -6 into destination object ‘dbuf’ of size 5152
        868 |         char dbuf[MAX_LONG_DOUBLE_CHARS+32];
      ```
      
      REASON: GCC-12 predicts that digits10() may return 9 or 10 through `return 9 + (v >= 1000000000UL)`.
      
      SOLUTION: add an assert to let the compiler know the possible length;
      
      4) redis-cli.c & redis-benchmark.c
      
      COMPILER: clang-14 with FORTIFY_SOURCE
      
      WARNING:
      ```
      redis-benchmark.c:1621:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL
      redis-cli.c:3015:2: warning: embedding a directive within macro arguments has undefined behavior [-Wembedded-directive] #ifdef USE_OPENSSL
      ```
      
      REASON: when _FORTIFY_SOURCE is enabled, the compiler will use the print() with
      check, which is a macro. this may result in the use of directives within the macro, which
      is undefined behavior.
      
      SOLUTION: move the directives-related code out of `print()`.
      
      5) server.c
      
      COMPILER: gcc-13 with FORTIFY_SOURCE
      
      WARNING:
      ```
      In function 'lookupCommandLogic',
          inlined from 'lookupCommandBySdsLogic' at server.c:3139:32:
      server.c:3102:66: error: '*(robj **)argv' may be used uninitialized [-Werror=maybe-uninitialized]
       3102 |     struct redisCommand *base_cmd = dictFetchValue(commands, argv[0]->ptr);
            |                                                              ~~~~^~~
      ```
      
      REASON: The compiler thinks that the `argc` returned by `sdssplitlen()` could be 0,
      resulting in an empty array of size 0 being passed to lookupCommandLogic.
      this should be a false positive, `argc` can't be 0 when strings are not NULL.
      
      SOLUTION: add an assert to let the compiler know that `argc` is positive.
      
      6) sha1.c
      
      COMPILER: gcc-12
      
      WARNING:
      ```
      In function ‘SHA1Update’,
          inlined from ‘SHA1Final’ at sha1.c:195:5:
      sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
        152 |             SHA1Transform(context->state, &data[i]);
            |             ^
      sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’
      sha1.c: In function ‘SHA1Final’:
      sha1.c:56:6: note: in a call to function ‘SHA1Transform’
         56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64])
            |      ^
      In function ‘SHA1Update’,
          inlined from ‘SHA1Final’ at sha1.c:198:9:
      sha1.c:152:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
        152 |             SHA1Transform(context->state, &data[i]);
            |             ^
      sha1.c:152:13: note: referencing argument 2 of type ‘const unsigned char[64]’
      sha1.c: In function ‘SHA1Final’:
      sha1.c:56:6: note: in a call to function ‘SHA1Transform’
         56 | void SHA1Transform(uint32_t state[5], const unsigned char buffer[64])
      ```
      
      REASON: due to the bug[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80922], when
      enable LTO, gcc-12 will not see `diagnostic ignored "-Wstringop-overread"`, resulting in a warning.
      
      SOLUTION: temporarily set SHA1Update to noinline to avoid compiler warnings due
      to LTO being enabled until the above gcc bug is fixed.
      
      7) zmalloc.h
      
      COMPILER: GCC-12
      
      WARNING: 
      ```
      In function ‘memset’,
          inlined from ‘moduleCreateContext’ at module.c:877:5,
          inlined from ‘RM_GetDetachedThreadSafeContext’ at module.c:8410:5:
      /usr/include/x86_64-linux-gnu/bits/string_fortified.h:59:10: warning: ‘__builtin_memset’ writing 104 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
         59 |   return __builtin___memset_chk (__dest, __ch, __len,
      ```
      
      REASON: due to the GCC-12 bug [https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96503],
      GCC-12 cannot see alloc_size, which causes GCC to think that the actual size of memory
      is 0 when checking with __glibc_objsize0().
      
      SOLUTION: temporarily set malloc-related interfaces to `noinline` to avoid compiler warnings
      due to LTO being enabled until the above gcc bug is fixed.
      
      ## Other changes
      1) Fixed `ps -p [pid]`  doesn't output `<defunct>` when using procps 4.x causing `replication
        child dies when parent is killed - diskless` test to fail.
      2) Add a new fortify CI with GCC-13 and ubuntu-lunar docker image.
      42c8c618
  4. 12 Apr, 2023 1 commit
    • Oran Agra's avatar
      Attempt to solve MacOS CI issues in GH Actions (#12013) · 997fa41e
      Oran Agra authored
      The MacOS CI in github actions often hangs without any logs. GH argues that
      it's due to resource utilization, either running out of disk space, memory, or CPU
      starvation, and thus the runner is terminated.
      
      This PR contains multiple attempts to resolve this:
      1. introducing pause_process instead of SIGSTOP, which waits for the process
        to stop before resuming the test, possibly resolving race conditions in some tests,
        this was a suspect since there was one test that could result in an infinite loop in that
       case, in practice this didn't help, but still a good idea to keep.
      2. disable the `save` config in many tests that don't need it, specifically ones that use
        heavy writes and could create large files.
      3. change the `populate` proc to use short pipeline rather than an infinite one.
      4. use `--clients 1` in the macos CI so that we don't risk running multiple resource
        demanding tests in parallel.
      5. enable `--verbose` to be repeated to elevate verbosity and print more info to stdout
        when a test or a server starts.
      997fa41e
  5. 15 Mar, 2023 1 commit
    • Binbin's avatar
      Fix WAITAOF reply when using last_offset and last_numreplicas (#11917) · 70b2c4f5
      Binbin authored
      WAITAOF wad added in #11713, its return is an array.
      But forget to handle WAITAOF in last_offset and last_numreplicas,
      causing WAITAOF to return a WAIT like reply.
      
      Tests was added to validate that case (both WAIT and WAITAOF).
      This PR also refactored processClientsWaitingReplicas a bit for better
      maintainability and readability.
      70b2c4f5
  6. 14 Mar, 2023 1 commit
    • Slava Koyfman's avatar
      Implementing the WAITAOF command (issue #10505) (#11713) · 9344f654
      Slava Koyfman authored
      
      
      Implementing the WAITAOF functionality which would allow the user to
      block until a specified number of Redises have fsynced all previous write
      commands to the AOF.
      
      Syntax: `WAITAOF <num_local> <num_replicas> <timeout>`
      Response: Array containing two elements: num_local, num_replicas
      num_local is always either 0 or 1 representing the local AOF on the master.
      num_replicas is the number of replicas that acknowledged the a replication
      offset of the last write being fsynced to the AOF.
      
      Returns an error when called on replicas, or when called with non-zero
      num_local on a master with AOF disabled, in all other cases the response
      just contains number of fsync copies.
      
      Main changes:
      * Added code to keep track of replication offsets that are confirmed to have
        been fsynced to disk.
      * Keep advancing master_repl_offset even when replication is disabled (and
        there's no replication backlog, only if there's an AOF enabled).
        This way we can use this command and it's mechanisms even when replication
        is disabled.
      * Extend REPLCONF ACK to `REPLCONF ACK <ofs> FACK <ofs>`, the FACK
        will be appended only if there's an AOF on the replica, and already ignored on
        old masters (thus backwards compatible)
      * WAIT now no longer wait for the replication offset after your last command, but
        rather the replication offset after your last write (or read command that caused
        propagation, e.g. lazy expiry).
      
      Unrelated changes:
      * WAIT command respects CLIENT_DENY_BLOCKING (not just CLIENT_MULTI)
      
      Implementation details:
      * Add an atomic var named `fsynced_reploff_pending` that's updated
        (usually by the bio thread) and later copied to the main `fsynced_reploff`
        variable (only if the AOF base file exists).
        I.e. during the initial AOF rewrite it will not be used as the fsynced offset
        since the AOF base is still missing.
      * Replace close+fsync bio job with new BIO_CLOSE_AOF (AOF specific)
        job that will also update fsync offset the field.
      * Handle all AOF jobs (BIO_CLOSE_AOF, BIO_AOF_FSYNC) in the same bio
        worker thread, to impose ordering on their execution. This solves a
        race condition where a job could set `fsynced_reploff_pending` to a higher
        value than another pending fsync job, resulting in indicating an offset
        for which parts of the data have not yet actually been fsynced.
        Imposing an ordering on the jobs guarantees that fsync jobs are executed
        in increasing order of replication offset.
      * Drain bio jobs when switching `appendfsync` to "always"
        This should prevent a write race between updates to `fsynced_reploff_pending`
        in the main thread (`flushAppendOnlyFile` when set to ALWAYS fsync), and
        those done in the bio thread.
      * Drain the pending fsync when starting over a new AOF to avoid race conditions
        with the previous AOF offsets overriding the new one (e.g. after switching to
        replicate from a new master).
      * Make sure to update the fsynced offset at the end of the initial AOF rewrite.
        a must in case there are no additional writes that trigger a periodic fsync,
        specifically for a replica that does a full sync.
      
      Limitations:
      It is possible to write a module and a Lua script that propagate to the AOF and doesn't
      propagate to the replication stream. see REDISMODULE_ARGV_NO_REPLICAS and luaRedisSetReplCommand.
      These features are incompatible with the WAITAOF command, and can result
      in two bad cases. The scenario is that the user executes command that only
      propagates to AOF, and then immediately
      issues a WAITAOF, and there's no further writes on the replication stream after that.
      1. if the the last thing that happened on the replication stream is a PING
        (which increased the replication offset but won't trigger an fsync on the replica),
        then the client would hang forever (will wait for an fack that the replica will never
        send sine it doesn't trigger any fsyncs).
      2. if the last thing that happened is a write command that got propagated properly,
        then WAITAOF will be released immediately, without waiting for an fsync (since
        the offset didn't change)
      
      Refactoring:
      * Plumbing to allow bio worker to handle multiple job types
        This introduces infrastructure necessary to allow BIO workers to
        not have a 1-1 mapping of worker to job-type. This allows in the
        future to assign multiple job types to a single worker, either as
        a performance/resource optimization, or as a way of enforcing
        ordering between specific classes of jobs.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      9344f654
  7. 22 Nov, 2022 1 commit
    • Binbin's avatar
      Fix set with duplicate elements causes sdiff to hang (#11530) · 3f8756a0
      Binbin authored
      
      
      This payload produces a set with duplicate elements (listpack encoding):
      ```
      restore _key 0 "\x14\x25\x25\x00\x00\x00\x0A\x00\x06\x01\x82\x5F\x35\x03\x04\x01\x82\x5F\x31\x03\x82\x5F\x33\x03\x00\x01\x82\x5F\x39\x03\x82\x5F\x33\x03\x08\x01\x02\x01\xFF\x0B\x00\x31\xBE\x7D\x41\x01\x03\x5B\xEC"
      
      smembers key
      1) "6"
      2) "_5"
      3) "4"
      4) "_1"
      5) "_3"  ---> dup
      6) "0"
      7) "_9"
      8) "_3"  ---> dup
      9) "8"
      10) "2"
      ```
      
      This kind of sets will cause SDIFF to hang, SDIFF generated a broken
      protocol and left the client hung. (Expected ten elements, but only
      got nine elements due to the duplication.)
      
      If we set `sanitize-dump-payload` to yes, we will be able to find
      the duplicate elements and report "ERR Bad data format".
      
      Discovered and discussed in #11290.
      
      This PR also improve prints when corrupt-dump-fuzzer hangs, it will
      print the cmds and the payload, an example like:
      ```
      Testing integration/corrupt-dump-fuzzer
      [TIMEOUT]: clients state report follows.
      sock6 => (SPAWNED SERVER) pid:28884
      Killing still running Redis server 28884
      commands caused test to hang:
      SDIFF __key 
      payload that caused test to hang: "\x14\balabala"
      ```
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      3f8756a0
  8. 12 Nov, 2022 1 commit
  9. 09 Nov, 2022 1 commit
    • Oran Agra's avatar
      diskless master, avoid bgsave child hung when fork parent crashes (#11463) · ccaef5c9
      Oran Agra authored
      During a diskless sync, if the master main process crashes, the child would
      have hung in `write`. This fix closes the read fd on the child side, so that if the
      parent crashes, the child will get a write error and exit.
      
      This change also fixes disk-based replication, BGSAVE and AOFRW.
      In that case the child wouldn't have been hang, it would have just kept
      running until done which may be pointless.
      
      There is a certain degree of risk here. in case there's a BGSAVE child that could
      maybe succeed and the parent dies for some reason, the old code would have let
      the child keep running and maybe succeed and avoid data loss.
      On the other hand, if the parent is restarted, it would have loaded an old rdb file
      (or none), and then the child could reach the end and rename the rdb file (data
      conflicting with what the parent has), or also have a race with another BGSAVE
      child that the new parent started.
      
      Note that i removed a comment saying a write error will be ignored in the child
      and handled by the parent (this comment was very old and i don't think relevant).
      ccaef5c9
  10. 19 Sep, 2022 1 commit
    • sundb's avatar
      Fix crash due to delete entry from compress quicklistNode and wrongly split quicklistNode (#11242) · 13d25dd9
      sundb authored
      This PR mainly deals with 2 crashes introduced in #9357,
      and fix the QUICKLIST-PACKED-THRESHOLD mess in external test mode.
      
      1. Fix crash due to deleting an entry from a compress quicklistNode
         When inserting a large element, we need to create a new quicklistNode first,
         and then delete its previous element, if the node where the deleted element is
         located is compressed, it will cause a crash.
         Now add `dont_compress` to quicklistNode, if we want to use a quicklistNode
         after some operation, we can use this flag like following:
      
          ```c
          node->dont_compress = 1; /* Prevent to be compressed */
          some_operation(node); /* This operation might try to compress this node */
          some_other_operation(node); /* We can use this node without decompress it */
          node->dont_compress = 0; /* Re-able compression */
          quicklistCompressNode(node);
          ```
      
         Perhaps in the future, we could just disable the current entry from being
         compressed during the iterator loop, but that would require more work.
      
      2. Fix crash due to wrongly split quicklist
         before #9357, the offset param of _quicklistSplitNode() will not negative.
         For now, when offset is negative, the split extent will be wrong.
         following example:
          ```c
          int orig_start = after ? offset + 1 : 0;
          int orig_extent = after ? -1 : offset;
          int new_start = after ? 0 : offset;
          int new_extent = after ? offset + 1 : -1;
          # offset: -2, after: 1, node->count: 2
          # current wrong range: [-1,-1] [0,-1]
          # correct range: [1,-1] [0, 1]
          ```
      
         Because only `_quicklistInsert()` splits the quicklistNode and only
         `quicklistInsertAfter()`, `quicklistInsertBefore()` call _quicklistInsert(), 
         so `quicklistReplaceEntry()` and `listTypeInsert()` might occur this crash.
         But the iterator of `listTypeInsert()` is alway from head to tail(iter->offset is
         always positive), so it is not affected.
         The final conclusion is this crash only occur when we insert a large element
         with negative index into a list, that affects `LSET` command and `RM_ListSet`
         module api.
           
      3. In external test mode, we need to restore quicklist packed threshold after
         when the end of test.
      4. Show `node->count` in quicklistRepr().
      5. Add new tcl proc `config_get_set` to support restoring config in tests.
      13d25dd9
  11. 24 Aug, 2022 1 commit
  12. 23 Aug, 2022 1 commit
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      
      
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wait for the "Server
        Initialized" message.
      * Fix issues with moduleconfigs test as a result from start_server
        waiting for "Server Initialized"
      * Fix issues with modules/infra test as a result of an additional module
        present
      
      Notes about Sentinel:
      Sentinel can't really rely on the tls module, since it uses hiredis to
      initiate connections and depends on OpenSSL (won't be able to use any
      other connection modules for that), so it was decided that when TLS is
      built as a module, sentinel does not support TLS at all.
      This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
      
      Example code of config in redis-tls.so(may be use in the future):
      RedisModuleString *tls_cfg = NULL;
      
      void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
          UNUSED(for_crash_report);
          RedisModule_InfoAddSection(ctx, "");
          RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
      }
      
      int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
      {
          if (argc != 2) return RedisModule_WrongArity(ctx);
          return RedisModule_ReplyWithString(ctx, argv[1]);
      }
      
      RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(privdata);
          return tls_cfg;
      }
      
      int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(err);
          REDISMODULE_NOT_USED(privdata);
          if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
          RedisModule_RetainString(NULL, new);
          tls_cfg = new;
          return REDISMODULE_OK;
      }
      
      int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
      {
          ....
          if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
              if (tls_cfg) {
                  RedisModule_FreeString(ctx, tls_cfg);
                  tls_cfg = NULL;
              }
              return REDISMODULE_ERR;
          }
          ...
      }
      Co-authored-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      4faddf18
  13. 12 Jul, 2022 1 commit
  14. 11 Jul, 2022 1 commit
    • Binbin's avatar
      Add cluster-port support to redis-cli --cluster (#10344) · 35e8ae3e
      Binbin authored
      
      
      In #9389, we add a new `cluster-port` config and make cluster bus port configurable,
      and currently redis-cli --cluster create/add-node doesn't support with a configurable `cluster-port` instance.
      Because redis-cli uses the old way (port + 10000) to send the `CLUSTER MEET` command.
      
      Now we add this support on redis-cli `--cluster`, note we don't need to explicitly pass in the
      `cluster-port` parameter, we can get the real `cluster-port` of the node in `clusterManagerNodeLoadInfo`,
      so the `--cluster create` and `--cluster add-node` interfaces have not changed.
      
      We will use the `cluster-port` when we are doing `CLUSTER MEET`, also note that `CLUSTER MEET` bus-port
      parameter was added in 4.0, so if the bus_port (the one in redis-cli) is 0, or equal (port + 10000),
      we just call `CLUSTER MEET` with 2 arguments, using the old form.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      35e8ae3e
  15. 26 Apr, 2022 1 commit
    • Madelyn Olson's avatar
      By default prevent cross slot operations in functions and scripts with # (#10615) · efcd1bf3
      Madelyn Olson authored
      Adds the `allow-cross-slot-keys` flag to Eval scripts and Functions to allow
      scripts to access keys from multiple slots.
      The default behavior is now that they are not allowed to do that (unlike before).
      This is a breaking change for 7.0 release candidates (to be part of 7.0.0), but
      not for previous redis releases since EVAL without shebang isn't doing this check.
      
      Note that the check is done on both the keys declared by the EVAL / FCALL command
      arguments, and also the ones used by the script when making a `redis.call`.
      
      A note about the implementation, there seems to have been some confusion
      about allowing access to non local keys. I thought I missed something in our
      wider conversation, but Redis scripts do block access to non-local keys.
      So the issue was just about cross slots being accessed.
      efcd1bf3
  16. 17 Apr, 2022 1 commit
    • guybe7's avatar
      Add RM_PublishMessageShard (#10543) · f49ff156
      guybe7 authored
      since PUBLISH and SPUBLISH use different dictionaries for channels and clients,
      and we already have an API for PUBLISH, it only makes sense to have one for SPUBLISH
      
      Add test coverage and unifying some test infrastructure.
      f49ff156
  17. 25 Mar, 2022 1 commit
    • zhaozhao.zz's avatar
      optimize(remove) usage of client's pending_querybuf (#10413) · 78bef6e1
      zhaozhao.zz authored
      To remove `pending_querybuf`, the key point is reusing `querybuf`, it means master client's `querybuf` is not only used to parse command, but also proxy to sub-replicas.
      
      1. add a new variable `repl_applied` for master client to record how many data applied (propagated via `replicationFeedStreamFromMasterStream()`) but not trimmed in `querybuf`.
      
      2. don't sdsrange `querybuf` in `commandProcessed()`, we trim it to `repl_applied` after the whole replication pipeline processed to avoid fragmented `sdsrange`. And here are some scenarios we cannot trim to `qb_pos`:
          * we don't receive complete command from master
          * master client blocked because of client pause
          * IO threads operate read, master client flagged with CLIENT_PENDING_COMMAND
      
          In these scenarios, `qb_pos` points to the part of the current command or the beginning of next command, and the current command is not applied yet, so the `repl_applied` is not equal to `qb_pos`.
      
      Some other notes:
      * Do not do big arg optimization on master client, since we can only sdsrange `querybuf` after data sent to replicas.
      * Set `qb_pos` and `repl_applied` to 0 when `freeClient` in `replicationCacheMaster`.
      * Rewrite `processPendingCommandsAndResetClient` to `processPendingCommandAndInputBuffer`, let `processInputBuffer` to be called successively after `processCommandAndResetClient`.
      78bef6e1
  18. 13 Feb, 2022 1 commit
    • Binbin's avatar
      Regression test for sync psync crash (#10288) · 62c8be28
      Binbin authored
      Added regression tests for #10020 / #10081 / #10243.
      The above PRs fixed some crashes due to an asserting,
      see function `clientHasPendingReplies` (introduced in #9166).
      
      This commit added some tests to cover the above scenario.
      These tests will all fail in #9166, althought fixed not,
      there is value in adding these tests to cover and verify
      the changes. And it also can cover #8868 (verify the logs).
      
      Other changes: 
      1. Reduces the wait time in `waitForBgsave` and `waitForBgrewriteaof`
      from 1s to 50ms, which should reduce the time for some tests.
      2. Improve the test infra to print context when `assert_match` fails.
      3. Improve the test infra to print `$error` when `assert_error` fails.
      ```
      Expected an error matching 'ERR*' but got 'OK' (context: type eval line 4 cmd {assert_error "ERR*" {r set a b}} proc ::test)
      ```
      62c8be28
  19. 05 Feb, 2022 1 commit
    • Jason Elbaum's avatar
      redis-cli generates command help tables from the results of COMMAND (#10043) · 5b17909c
      Jason Elbaum authored
      
      
      This is a followup to #9656 and implements the following step mentioned in that PR:
      
      * When possible, extract all the help and completion tips from COMMAND DOCS (Redis 7.0 and up)
      * If COMMAND DOCS fails, use the static help.h compiled into redis-cli.
      * Supplement additional command names from COMMAND (pre-Redis 7.0)
      
      The last step is needed to add module command and other non-standard commands.
      
      This PR does not change the interactive hinting mechanism, which still uses only the param
      strings to provide somewhat unreliable and inconsistent command hints (see #8084).
      That task is left for a future PR. 
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5b17909c
  20. 17 Jan, 2022 1 commit
    • Oran Agra's avatar
      Set repl-diskless-sync to yes by default, add repl-diskless-sync-max-replicas (#10092) · ae899589
      Oran Agra authored
      1. enable diskless replication by default
      2. add a new config named repl-diskless-sync-max-replicas that enables
         replication to start before the full repl-diskless-sync-delay was
         reached.
      3. put replica online sooner on the master (see below)
      4. test suite uses repl-diskless-sync-delay of 0 to be faster
      5. a few tests that use multiple replica on a pre-populated master, are
         now using the new repl-diskless-sync-max-replicas
      6. fix possible timing issues in a few cluster tests (see below)
      
      put replica online sooner on the master 
      ----------------------------------------------------
      there were two tests that failed because they needed for the master to
      realize that the replica is online, but the test code was actually only
      waiting for the replica to realize it's online, and in diskless it could
      have been before the master realized it.
      
      changes include two things:
      1. the tests wait on the right thing
      2. issues in the master, putting the replica online in two steps.
      
      the master used to put the replica as online in 2 steps. the first
      step was to mark it as online, and the second step was to enable the
      write event (only after getting ACK), but in fact the first step didn't
      contains some of the tasks to put it online (like updating good slave
      count, and sending the module event). this meant that if a test was
      waiting to see that the replica is online form the point of view of the
      master, and then confirm that the module got an event, or that the
      master has enough good replicas, it could fail due to timing issues.
      
      so now the full effect of putting the replica online, happens at once,
      and only the part about enabling the writes is delayed till the ACK.
      
      fix cluster tests 
      --------------------
      I added some code to wait for the replica to sync and avoid race
      conditions.
      later realized the sentinel and cluster tests where using the original 5
      seconds delay, so changed it to 0.
      
      this means the other changes are probably not needed, but i suppose
      they're still better (avoid race conditions)
      ae899589
  21. 05 Jan, 2022 1 commit
    • filipe oliveira's avatar
      Added INFO LATENCYSTATS section: latency by percentile distribution/latency by... · 5dd15443
      filipe oliveira authored
      
      Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462)
      
      # Short description
      
      The Redis extended latency stats track per command latencies and enables:
      - exporting the per-command percentile distribution via the `INFO LATENCYSTATS` command.
        **( percentile distribution is not mergeable between cluster nodes ).**
      - exporting the per-command cumulative latency distributions via the `LATENCY HISTOGRAM` command.
        Using the cumulative distribution of latencies we can merge several stats from different cluster nodes
        to calculate aggregate metrics .
      
      By default, the extended latency monitoring is enabled since the overhead of keeping track of the
      command latency is very small.
       
      If you don't want to track extended latency metrics, you can easily disable it at runtime using the command:
       - `CONFIG SET latency-tracking no`
      
      By default, the exported latency percentiles are the p50, p99, and p999.
      You can alter them at runtime using the command:
      - `CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"`
      
      
      ## Some details:
      - The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command
        was called for the first time.
      - With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable
        ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable
        vs this branch. 
      - We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf )
      
      ## `INFO LATENCYSTATS` exposition format
      
         - Format: `latency_percentiles_usec_<CMDNAME>:p0=XX,p50....` 
      
      ## `LATENCY HISTOGRAM [command ...]` exposition format
      
      Return a cumulative distribution of latencies in the format of a histogram for the specified command names.
      
      The histogram is composed of a map of time buckets:
      - Each representing a latency range, between 1 nanosecond and roughly 1 second.
      - Each bucket covers twice the previous bucket's range.
      - Empty buckets are not printed.
      - Everything above 1 sec is considered +Inf.
      - At max there will be log2(1000000000)=30 buckets
      
      We reply a map for each command in the format:
      `<command name> : { `calls`: <total command calls> , `histogram` : { <bucket 1> : latency , < bucket 2> : latency, ...  } }`
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5dd15443
  22. 03 Jan, 2022 1 commit
    • chenyang8094's avatar
      Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788) · 87789fae
      chenyang8094 authored
      
      
      Implement Multi-Part AOF mechanism to avoid overheads during AOFRW.
      Introducing a folder with multiple AOF files tracked by a manifest file.
      
      The main issues with the the original AOFRW mechanism are:
      * buffering of commands that are processed during rewrite (consuming a lot of RAM)
      * freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.
      * double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)
      
      The main modifications of this PR:
      1. Remove the AOF rewrite buffer and related code.
      2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type,
        it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only
        one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the
        incremental commands since the last AOFRW.
      3. Use a AOF manifest file to record and manage these AOF files mentioned above.
      4. The original configuration of `appendfilename` will be the base part of the new file name, for example:
        `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof`
      5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename`
      6. Remove the `aof_rewrite_buffer_length` field in info.
      7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs.
        It also gives users the opportunity to preserve the history AOFs. just for testing use now.
      8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now),
        we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be
        delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit
        period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately.
      9. Support upgrade (load) data from old version redis.
      10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and
        manifest file will be placed in this directory.
      11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if
        `aof-load-truncated` is enabled.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      87789fae
  23. 02 Jan, 2022 1 commit
    • Viktor Söderqvist's avatar
      Wait for replicas when shutting down (#9872) · 45a155bd
      Viktor Söderqvist authored
      
      
      To avoid data loss, this commit adds a grace period for lagging replicas to
      catch up the replication offset.
      
      Done:
      
      * Wait for replicas when shutdown is triggered by SIGTERM and SIGINT.
      
      * Wait for replicas when shutdown is triggered by the SHUTDOWN command. A new
        blocked client type BLOCKED_SHUTDOWN is introduced, allowing multiple clients
        to call SHUTDOWN in parallel.
        Note that they don't expect a response unless an error happens and shutdown is aborted.
      
      * Log warning for each replica lagging behind when finishing shutdown.
      
      * CLIENT_PAUSE_WRITE while waiting for replicas.
      
      * Configurable grace period 'shutdown-timeout' in seconds (default 10).
      
      * New flags for the SHUTDOWN command:
      
          - NOW disables the grace period for lagging replicas.
      
          - FORCE ignores errors writing the RDB or AOF files which would normally
            prevent a shutdown.
      
          - ABORT cancels ongoing shutdown. Can't be combined with other flags.
      
      * New field in the output of the INFO command: 'shutdown_in_milliseconds'. The
        value is the remaining maximum time to wait for lagging replicas before
        finishing the shutdown. This field is present in the Server section **only**
        during shutdown.
      
      Not directly related:
      
      * When shutting down, if there is an AOF saving child, it is killed **even** if AOF
        is disabled. This can happen if BGREWRITEAOF is used when AOF is off.
      
      * Client pause now has end time and type (WRITE or ALL) per purpose. The
        different pause purposes are *CLIENT PAUSE command*, *failover* and
        *shutdown*. If clients are unpaused for one purpose, it doesn't affect client
        pause for other purposes. For example, the CLIENT UNPAUSE command doesn't
        affect client pause initiated by the failover or shutdown procedures. A completed
        failover or a failed shutdown doesn't unpause clients paused by the CLIENT
        PAUSE command.
      
      Notes:
      
      * DEBUG RESTART doesn't wait for replicas.
      
      * We already have a warning logged when a replica disconnects. This means that
        if any replica connection is lost during the shutdown, it is either logged as
        disconnected or as lagging at the time of exit.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      45a155bd
  24. 27 Dec, 2021 1 commit
  25. 19 Dec, 2021 2 commits
    • Oran Agra's avatar
      Add external test that runs without debug command (#9964) · 6add1b72
      Oran Agra authored
      - add needs:debug flag for some tests
      - disable "save" in external tests (speedup?)
      - use debug_digest proc instead of debug command directly so it can be skipped
      - use OBJECT ENCODING instead of DEBUG OBJECT to get encoding
      - add a proc for OBJECT REFCOUNT so it can be skipped
      - move a bunch of tests in latency_monitor tests to happen later so that latency monitor has some values in it
      - add missing close_replication_stream calls
      - make sure to close the temp client if DEBUG LOG fails
      6add1b72
    • YaacovHazan's avatar
      Protected configs and sensitive commands (#9920) · ae2f5b7b
      YaacovHazan authored
      Block sensitive configs and commands by default.
      
      * `enable-protected-configs` - block modification of configs with the new `PROTECTED_CONFIG` flag.
         Currently we add this flag to `dbfilename`, and `dir` configs,
         all of which are non-mutable configs that can set a file redis will write to.
      * `enable-debug-command` - block the `DEBUG` command
      * `enable-module-command` - block the `MODULE` command
      
      These have a default value set to `no`, so that these features are not
      exposed by default to client connections, and can only be set by modifying the config file.
      
      Users can change each of these to either `yes` (allow all access), or `local` (allow access from
      local TCP connections and unix domain connections)
      
      Note that this is a **breaking change** (specifically the part about MODULE command being disabled by default).
      I.e. we don't consider DEBUG command being blocked as an issue (people shouldn't have been using it),
      and the few configs we protected are unlikely to have been set at runtime anyway.
      On the other hand, it's likely to assume some users who use modules, load them from the config file anyway.
      Note that's the whole point of this PR, for redis to be more secure by default and reduce the attack surface on
      innocent users, so secure defaults will necessarily mean a breaking change.
      ae2f5b7b
  26. 17 Dec, 2021 1 commit
    • ny0312's avatar
      Introduce memory management on cluster link buffers (#9774) · 792afb44
      ny0312 authored
      Introduce memory management on cluster link buffers:
       * Introduce a new `cluster-link-sendbuf-limit` config that caps memory usage of cluster bus link send buffers.
       * Introduce a new `CLUSTER LINKS` command that displays current TCP links to/from peers.
       * Introduce a new `mem_cluster_links` field under `INFO` command output, which displays the overall memory usage by all current cluster links.
       * Introduce a new `total_cluster_links_buffer_limit_exceeded` field under `CLUSTER INFO` command output, which displays the accumulated count of cluster links freed due to `cluster-link-sendbuf-limit`.
      792afb44
  27. 13 Dec, 2021 1 commit
    • yoav-steinberg's avatar
      Fix possible int overflow when hashing an sds. (#9916) · c7dc17fc
      yoav-steinberg authored
      This caused a crash when adding elements larger than 2GB to a set (same goes for hash keys). See #8455.
      
      Details:
      * The fix makes the dict hash functions receive a `size_t` instead of an `int`. In practice the dict hash functions
        call siphash which receives a `size_t` and the callers to the hash function pass a `size_t` to it so the fix is trivial.
      * The issue was recreated by attempting to add a >2gb value to a set. Appropriate tests were added where I create
        a set with large elements and check basic functionality on it (SADD, SCARD, SPOP, etc...).
      * When I added the tests I also refactored a bit all the tests code which is run under the `--large-memory` flag.
        This removed code duplication for the test framework's `write_big_bulk` and `write_big_bulk` code and also takes
        care of not allocating the test frameworks helper huge string used by these tests when not run under `--large-memory`.
      * I also added the _violoations.tcl_ unit tests to be part of the entire test suite and leaned up non relevant list related
        tests that were in there. This was done in this PR because most of the _violations_ tests are "large memory" tests.
      c7dc17fc
  28. 18 Nov, 2021 1 commit
    • guybe7's avatar
      Obliterate STRALGO! add LCS (which only works on keys) (#9799) · af748988
      guybe7 authored
      Drop the STRALGO command, now LCS is a command of its own and it only works on keys (not input strings).
      The motivation is that STRALGO's syntax was really messed-up...
      - assumes all (future) string algorithms will take similar arguments
      - mixes command that takes keys and one that doesn't in the same command.
      - make it nearly impossible to expose the right key spec in COMMAND INFO (issues cluster clients)
      - hard for cluster clients to determine the key names (firstkey, lastkey, etc)
      - hard for ACL / flags (is it a read command?)
      
      This is a breaking change.
      af748988
  29. 11 Nov, 2021 2 commits
    • Ozan Tezcan's avatar
      Add sanitizer support and clean up sanitizer findings (#9601) · b91d8b28
      Ozan Tezcan authored
      - Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
      - To build Redis with desired sanitizer : `make SANITIZER=undefined`
      - There were some sanitizer findings, cleaned up codebase
      - Added tests with address and undefined behavior sanitizers to daily CI.
      - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).
      
      Basically, there are three types of issues : 
      
      **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
      does not support unaligned access. Redis does unaligned access only on supported platforms.
      
      **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
       **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit. 
      b91d8b28
    • yoav-steinberg's avatar
      Archive external redis log in external tests (#9765) · cd6b3d55
      yoav-steinberg authored
      On test failure store the external redis server logs as CI artifacts so we can review them.
      
      Write test name to server log for external server tests.
      This is attempted and silently failed in case external server doesn't support it.
      Note that in non-external server mode we use a more robust method of writing to the log which doesn't depend on the
      server actually running/working. This isn't possible for externl servers and required for some complex tests which are
      skipped in external mode anyway.
      
      Cleanup: remove dup code.
      cd6b3d55
  30. 07 Nov, 2021 1 commit
  31. 02 Nov, 2021 1 commit
    • Oran Agra's avatar
      attempt to fix tracking test issue with external tests due to lazy free (#9722) · 87321deb
      Oran Agra authored
      The External tests started failing recently for unclear reason:
      ```
      *** [err]: Tracking invalidation message of eviction keys should be before response in tests/unit/tracking.tcl
      Expected '0' to be equal to 'invalidate volatile-key' (context: type eval line 21 cmd {assert_equal $res {invalidate volatile-key}} proc ::test)
      ```
      
      I suspect the issue is that the used_memory sample is taken while a lazy free is still being processed.
      87321deb
  32. 24 Sep, 2021 1 commit
  33. 23 Sep, 2021 1 commit
    • yoav-steinberg's avatar
      Client eviction (#8687) · 2753429c
      yoav-steinberg authored
      
      
      ### Description
      A mechanism for disconnecting clients when the sum of all connected clients is above a
      configured limit. This prevents eviction or OOM caused by accumulated used memory
      between all clients. It's a complimentary mechanism to the `client-output-buffer-limit`
      mechanism which takes into account not only a single client and not only output buffers
      but rather all memory used by all clients.
      
      #### Design
      The general design is as following:
      * We track memory usage of each client, taking into account all memory used by the
        client (query buffer, output buffer, parsed arguments, etc...). This is kept up to date
        after reading from the socket, after processing commands and after writing to the socket.
      * Based on the used memory we sort all clients into buckets. Each bucket contains all
        clients using up up to x2 memory of the clients in the bucket below it. For example up
        to 1m clients, up to 2m clients, up to 4m clients, ...
      * Before processing a command and before sleep we check if we're over the configured
        limit. If we are we start disconnecting clients from larger buckets downwards until we're
        under the limit.
      
      #### Config
      `maxmemory-clients` max memory all clients are allowed to consume, above this threshold
      we disconnect clients.
      This config can either be set to 0 (meaning no limit), a size in bytes (possibly with MB/GB
      suffix), or as a percentage of `maxmemory` by using the `%` suffix (e.g. setting it to `10%`
      would mean 10% of `maxmemory`).
      
      #### Important code changes
      * During the development I encountered yet more situations where our io-threads access
        global vars. And needed to fix them. I also had to handle keeps the clients sorted into the
        memory buckets (which are global) while their memory usage changes in the io-thread.
        To achieve this I decided to simplify how we check if we're in an io-thread and make it
        much more explicit. I removed the `CLIENT_PENDING_READ` flag used for checking
        if the client is in an io-thread (it wasn't used for anything else) and just used the global
        `io_threads_op` variable the same way to check during writes.
      * I optimized the cleanup of the client from the `clients_pending_read` list on client freeing.
        We now store a pointer in the `client` struct to this list so we don't need to search in it
        (`pending_read_list_node`).
      * Added `evicted_clients` stat to `INFO` command.
      * Added `CLIENT NO-EVICT ON|OFF` sub command to exclude a specific client from the
        client eviction mechanism. Added corrosponding 'e' flag in the client info string.
      * Added `multi-mem` field in the client info string to show how much memory is used up
        by buffered multi commands.
      * Client `tot-mem` now accounts for buffered multi-commands, pubsub patterns and
        channels (partially), tracking prefixes (partially).
      * CLIENT_CLOSE_ASAP flag is now handled in a new `beforeNextClient()` function so
        clients will be disconnected between processing different clients and not only before sleep.
        This new function can be used in the future for work we want to do outside the command
        processing loop but don't want to wait for all clients to be processed before we get to it.
        Specifically I wanted to handle output-buffer-limit related closing before we process client
        eviction in case the two race with each other.
      * Added a `DEBUG CLIENT-EVICTION` command to print out info about the client eviction
        buckets.
      * Each client now holds a pointer to the client eviction memory usage bucket it belongs to
        and listNode to itself in that bucket for quick removal.
      * Global `io_threads_op` variable now can contain a `IO_THREADS_OP_IDLE` value
        indicating no io-threading is currently being executed.
      * In order to track memory used by each clients in real-time we can't rely on updating
        these stats in `clientsCron()` alone anymore. So now I call `updateClientMemUsage()`
        (used to be `clientsCronTrackClientsMemUsage()`) after command processing, after
        writing data to pubsub clients, after writing the output buffer and after reading from the
        socket (and maybe other places too). The function is written to be fast.
      * Clients are evicted if needed (with appropriate log line) in `beforeSleep()` and before
        processing a command (before performing oom-checks and key-eviction).
      * All clients memory usage buckets are grouped as follows:
        * All clients using less than 64k.
        * 64K..128K
        * 128K..256K
        * ...
        * 2G..4G
        * All clients using 4g and up.
      * Added client-eviction.tcl with a bunch of tests for the new mechanism.
      * Extended maxmemory.tcl to test the interaction between maxmemory and
        maxmemory-clients settings.
      * Added an option to flag a numeric configuration variable as a "percent", this means that
        if we encounter a '%' after the number in the config file (or config set command) we
        consider it as valid. Such a number is store internally as a negative value. This way an
        integer value can be interpreted as either a percent (negative) or absolute value (positive).
        This is useful for example if some numeric configuration can optionally be set to a percentage
        of something else.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2753429c
  34. 12 Sep, 2021 1 commit
    • Huang Zhw's avatar
      bitpos/bitcount add bit index (#9324) · 75dd2309
      Huang Zhw authored
      Make bitpos/bitcount support bit index:
      
      ```
      BITPOS key bit [start [end [BIT|BYTE]]]
      BITCOUNT key [start end [BIT|BYTE]]
      ```
      
      The default behavior is `BYTE`, so these commands are still compatible with old.
      75dd2309
  35. 05 Aug, 2021 1 commit
    • Oran Agra's avatar
      corrupt-dump-fuzzer test, avoid creating junk keys (#9302) · 3f3f678a
      Oran Agra authored
      The execution of the RPOPLPUSH command by the fuzzer created junk keys,
      that were later being selected by RANDOMKEY and modified.
      This also meant that lists were statistically tested more than other
      files.
      
      Fix the fuzzer not to pass junk key names to RPOPLPUSH, and add a check
      that detects that new keys are not added by the fuzzer to detect future
      similar issues.
      3f3f678a
  36. 29 Jul, 2021 1 commit
    • Wen Hui's avatar
      Remove duplicate zero-port sentinels (#9240) · db415364
      Wen Hui authored
      The issue is that when a sentinel with the same address and IP is turned on with a different runid, its port is set to 0 but it is still present in the dictionary master->sentinels which contain all the sentinels for a master.
      
      This causes a problem when we do INFO SENTINEL because it takes the size of the dictionary of sentinels. This might also cause a problem for failover if enough sentinels have their port set to 0 since the number of voters in failover is also determined by the size of the dictionary of sentinels.
      
      This commits removes the sentinels with the port set to zero from the dictionary of sentinels.
      Fixes #8786
      db415364
  37. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  38. 09 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve test suite to handle external servers better. (#9033) · 8a86bca5
      Yossi Gottlieb authored
      This commit revives the improves the ability to run the test suite against
      external servers, instead of launching and managing `redis-server` processes as
      part of the test fixture.
      
      This capability existed in the past, using the `--host` and `--port` options.
      However, it was quite limited and mostly useful when running a specific tests.
      Attempting to run larger chunks of the test suite experienced many issues:
      
      * Many tests depend on being able to start and control `redis-server` themselves,
      and there's no clear distinction between external server compatible and other
      tests.
      * Cluster mode is not supported (resulting with `CROSSSLOT` errors).
      
      This PR cleans up many things and makes it possible to run the entire test suite
      against an external server. It also provides more fine grained controls to
      handle cases where the external server supports a subset of the Redis commands,
      limited number of databases, cluster mode, etc.
      
      The tests directory now contains a `README.md` file that describes how this
      works.
      
      This commit also includes additional cleanups and fixes:
      
      * Tests can now be tagged.
      * Tag-based selection is now unified across `start_server`, `tags` and `test`.
      * More information is provided about skipped or ignored tests.
      * Repeated patterns in tests have been extracted to common procedures, both at a
        global level and on a per-test file basis.
      * Cleaned up some cases where test setup was based on a previous test executing
        (a major anti-pattern that repeats itself in many places).
      * Cleaned up some cases where test teardown was not part of a test (in the
        future we should have dedicated teardown code that executes even when tests
        fail).
      * Fixed some tests that were flaky running on external servers.
      8a86bca5