1. 23 Aug, 2022 1 commit
    • Oran Agra's avatar
      Build TLS as a loadable module · 4faddf18
      Oran Agra authored
      
      
      * Support BUILD_TLS=module to be loaded as a module via config file or
        command line. e.g. redis-server --loadmodule redis-tls.so
      * Updates to redismodule.h to allow it to be used side by side with
        server.h by defining REDISMODULE_CORE_MODULE
      * Changes to server.h, redismodule.h and module.c to avoid repeated
        type declarations (gcc 4.8 doesn't like these)
      * Add a mechanism for non-ABI neutral modules (ones who include
        server.h) to refuse loading if they detect not being built together with
        redis (release.c)
      * Fix wrong signature of RedisModuleDefragFunc, this could break
        compilation of a module, but not the ABI
      * Move initialization of listeners in server.c to be after loading
        the modules
      * Config TLS after initialization of listeners
      * Init cluster after initialization of listeners
      * Add TLS module to CI
      * Fix a test suite race conditions:
        Now that the listeners are initialized later, it's not sufficient to
        wait for the PID message in the log, we need to wait for the "Server
        Initialized" message.
      * Fix issues with moduleconfigs test as a result from start_server
        waiting for "Server Initialized"
      * Fix issues with modules/infra test as a result of an additional module
        present
      
      Notes about Sentinel:
      Sentinel can't really rely on the tls module, since it uses hiredis to
      initiate connections and depends on OpenSSL (won't be able to use any
      other connection modules for that), so it was decided that when TLS is
      built as a module, sentinel does not support TLS at all.
      This means that it keeps using redis_tls_ctx and redis_tls_client_ctx directly.
      
      Example code of config in redis-tls.so(may be use in the future):
      RedisModuleString *tls_cfg = NULL;
      
      void tlsInfo(RedisModuleInfoCtx *ctx, int for_crash_report) {
          UNUSED(for_crash_report);
          RedisModule_InfoAddSection(ctx, "");
          RedisModule_InfoAddFieldLongLong(ctx, "var", 42);
      }
      
      int tlsCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
      {
          if (argc != 2) return RedisModule_WrongArity(ctx);
          return RedisModule_ReplyWithString(ctx, argv[1]);
      }
      
      RedisModuleString *getStringConfigCommand(const char *name, void *privdata) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(privdata);
          return tls_cfg;
      }
      
      int setStringConfigCommand(const char *name, RedisModuleString *new, void *privdata, RedisModuleString **err) {
          REDISMODULE_NOT_USED(name);
          REDISMODULE_NOT_USED(err);
          REDISMODULE_NOT_USED(privdata);
          if (tls_cfg) RedisModule_FreeString(NULL, tls_cfg);
          RedisModule_RetainString(NULL, new);
          tls_cfg = new;
          return REDISMODULE_OK;
      }
      
      int RedisModule_OnLoad(void *ctx, RedisModuleString **argv, int argc)
      {
          ....
          if (RedisModule_CreateCommand(ctx,"tls",tlsCommand,"",0,0,0) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_RegisterStringConfig(ctx, "cfg", "", REDISMODULE_CONFIG_DEFAULT, getStringConfigCommand, setStringConfigCommand, NULL, NULL) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      
          if (RedisModule_LoadConfigs(ctx) == REDISMODULE_ERR) {
              if (tls_cfg) {
                  RedisModule_FreeString(ctx, tls_cfg);
                  tls_cfg = NULL;
              }
              return REDISMODULE_ERR;
          }
          ...
      }
      Co-authored-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      4faddf18
  2. 21 Aug, 2022 1 commit
  3. 18 Aug, 2022 4 commits
    • Binbin's avatar
      Fix CLUSTERDOWN issue in cluster reshard unblock test (#11139) · 3a16ad30
      Binbin authored
      change the cluster-node-timeout from 1 to 1000
      3a16ad30
    • guybe7's avatar
      Repurpose redisCommandArg's name as the unique ID (#11051) · 223046ec
      guybe7 authored
      This PR makes sure that "name" is unique for all arguments in the same
      level (i.e. all args of a command and all args within a block/oneof).
      This means several argument with identical meaning can be referred to together,
      but also if someone needs to refer to a specific one, they can use its full path.
      
      In addition, the "display_text" field has been added, to be used by redis.io
      in order to render the syntax of the command (for the vast majority it is
      identical to "name" but sometimes we want to use a different string
      that is not "name")
      The "display" field is exposed via COMMAND DOCS and will be present
      for every argument, except "oneof" and "block" (which are container
      arguments)
      
      Other changes:
      1. Make sure we do not have any container arguments ("oneof" or "block")
         that contain less than two sub-args (otherwise it doesn't make sense)
      2. migrate.json: both AUTH and AUTH2 should not be "optional"
      3. arg names cannot contain underscores, and force the usage of hyphens
        (most of these were a result of the script that generated the initial json files
        from redis.io commands.json). 
      223046ec
    • Binbin's avatar
      Fix memory leak in moduleFreeCommand (#11147) · fc3956e8
      Binbin authored
      Currently, we call zfree(cmd->args), but the argument array
      needs to be freed recursively (there might be sub-args).
      Also fixed memory leaks on cmd->tips and cmd->history.
      
      Fixes #11145
      fc3956e8
    • Meir Shpilraien (Spielrein)'s avatar
      Fix replication inconsistency on modules that uses key space notifications (#10969) · 508a1388
      Meir Shpilraien (Spielrein) authored
      Fix replication inconsistency on modules that uses key space notifications.
      
      ### The Problem
      
      In general, key space notifications are invoked after the command logic was
      executed (this is not always the case, we will discuss later about specific
      command that do not follow this rules). For example, the `set x 1` will trigger
      a `set` notification that will be invoked after the `set` logic was performed, so
      if the notification logic will try to fetch `x`, it will see the new data that was written.
      Consider the scenario on which the notification logic performs some write
      commands. for example, the notification logic increase some counter,
      `incr x{counter}`, indicating how many times `x` was changed.
      The logical order by which the logic was executed is has follow:
      
      ```
      set x 1
      incr x{counter}
      ```
      
      The issue is that the `set x 1` command is added to the replication buffer
      at the end of the command invocation (specifically after the key space
      notification logic was invoked and performed the `incr` command).
      The replication/aof sees the commands in the wrong order:
      
      ```
      incr x{counter}
      set x 1
      ```
      
      In this specific example the order is less important.
      But if, for example, the notification would have deleted `x` then we would
      end up with primary-replica inconsistency.
      
      ### The Solution
      
      Put the command that cause the notification in its rightful place. In the
      above example, the `set x 1` command logic was executed before the
      notification logic, so it should be added to the replication buffer before
      the commands that is invoked by the notification logic. To achieve this,
      without a major code refactoring, we save a placeholder in the replication
      buffer, when finishing invoking the command logic we check if the command
      need to be replicated, and if it does, we use the placeholder to add it to the
      replication buffer instead of appending it to the end.
      
      To be efficient and not allocating memory on each command to save the
      placeholder, the replication buffer array was modified to reuse memory
      (instead of allocating it each time we want to replicate commands).
      Also, to avoid saving a placeholder when not needed, we do it only for
      WRITE or MAY_REPLICATE commands.
      
      #### Additional Fixes
      
      * Expire and Eviction notifications:
        * Expire/Eviction logical order was to first perform the Expire/Eviction
          and then the notification logic. The replication buffer got this in the
          other way around (first notification effect and then the `del` command).
          The PR fixes this issue.
        * The notification effect and the `del` command was not wrap with
          `multi-exec` (if needed). The PR also fix this issue.
      * SPOP command:
        * On spop, the `spop` notification was fired before the command logic
          was executed. The change in this PR would have cause the replication
          order to be change (first `spop` command and then notification `logic`)
          although the logical order is first the notification logic and then the
          `spop` logic. The right fix would have been to move the notification to
          be fired after the command was executed (like all the other commands),
          but this can be considered a breaking change. To overcome this, the PR
          keeps the current behavior and changes the `spop` code to keep the right
          logical order when pushing commands to the replication buffer. Another PR
          will follow to fix the SPOP properly and match it to the other command (we
          split it to 2 separate PR's so it will be easy to cherry-pick this PR to 7.0 if
          we chose to).
      
      #### Unhanded Known Limitations
      
      * key miss event:
        * On key miss event, if a module performed some write command on the
          event (using `RM_Call`), the `dirty` counter would increase and the read
          command that cause the key miss event would be replicated to the replication
          and aof. This problem can also happened on a write command that open
          some keys but eventually decides not to perform any action. We decided
          not to handle this problem on this PR because the solution is complex
          and will cause additional risks in case we will want to cherry-pick this PR.
          We should decide if we want to handle it in future PR's. For now, modules
          writers is advice not to perform any write commands on key miss event.
      
      #### Testing
      
      * We already have tests to cover cases where a notification is invoking write
        commands that are also added to the replication buffer, the tests was modified
        to verify that the replica gets the command in the correct logical order.
      * Test was added to verify that `spop` behavior was kept unchanged.
      * Test was added to verify key miss event behave as expected.
      * Test was added to verify the changes do not break lazy expiration.
      
      #### Additional Changes
      
      * `propagateNow` function can accept a special dbid, -1, indicating not
        to replicate `select`. We use this to replicate `multi/exec` on `propagatePendingCommands`
        function. The side effect of this change is that now the `select` command
        will appear inside the `multi/exec` block on the replication stream (instead of
        outside of the `multi/exec` block). Tests was modified to match this new behavior.
      508a1388
  4. 16 Aug, 2022 1 commit
  5. 14 Aug, 2022 2 commits
    • guybe7's avatar
      Rename offset and xsetid tags (#11103) · 1189680e
      guybe7 authored
      There's really no point in having dedicated flags to test these features
      (why shouldn't all commands/features get their own tag?)
      1189680e
    • sundb's avatar
      Add missing lua_pop in luaGetFromRegistry (#11097) · 8aad2ac3
      sundb authored
      This pr mainly has the following four changes:
      
      1. Add missing lua_pop in `luaGetFromRegistry`.
          This bug affects `redis.register_function`, where `luaGetFromRegistry` in
          `luaRegisterFunction` will return null when we call `redis.register_function` nested.
          .e.g
          ```
          FUNCTION LOAD "#!lua name=mylib \n local lib=redis \n lib.register_function('f2', function(keys, args) lib.register_function('f1', function () end) end)"
          fcall f2 0
          ````
          But since we exit when luaGetFromRegistry returns null, it does not cause the stack to grow indefinitely.
      
      3. When getting `REGISTRY_RUN_CTX_NAME` from the registry, use `serverAssert`
          instead of error return. Since none of these lua functions are registered at the time
          of function load, scriptRunCtx will never be NULL.
      4. Add `serverAssert` for `luaLdbLineHook`, `luaEngineLoadHook`.
      5. Remove `luaGetFromRegistry` from `redis_math_random` and
          `redis_math_randomseed`, it looks like they are redundant.
      8aad2ac3
  6. 10 Aug, 2022 1 commit
  7. 07 Aug, 2022 2 commits
  8. 04 Aug, 2022 1 commit
  9. 03 Aug, 2022 2 commits
  10. 02 Aug, 2022 1 commit
    • Binbin's avatar
      Solve usleep compilation warning in keyspace_events.c (#11073) · 9f0f533b
      Binbin authored
      There is a -Wimplicit-function-declaration warning in here:
      ```
      keyspace_events.c: In function ‘KeySpace_NotificationGeneric’:
      keyspace_events.c:67:9: warning: implicit declaration of function ‘usleep’; did you mean ‘sleep’? [-Wimplicit-function-declaration]
         67 |         usleep(1);
            |         ^~~~~~
            |         sleep
      ```
      9f0f533b
  11. 01 Aug, 2022 1 commit
  12. 31 Jul, 2022 1 commit
  13. 27 Jul, 2022 2 commits
    • Binbin's avatar
      Fix bgsaveerr issue in psync wrong offset test (#11043) · e7144693
      Binbin authored
      The kill above is sometimes successful and sometimes already too late.
      The PING in pysnc wrong offset test got rejected by bgsaveerr because
      lastbgsave_status is C_ERR.
      
      In theory, using diskless can avoid PING being affected, because when
      the replica is dropped, we will kill the child with SIGUSR1, and this
      will not affect lastbgsave_status.
      
      Anyway, this kill is not particularly needed here, dropping the kill
      is the best one, since we do have the waitForBgsave, so just let it
      take care of the bgsave. No need for fast termination.
      e7144693
    • guybe7's avatar
      Adds RM_Microseconds and RM_CachedMicroseconds (#11016) · 45c99d70
      guybe7 authored
      RM_Microseconds
      Return the wall-clock Unix time, in microseconds
      
      RM_CachedMicroseconds
      Returns a cached copy of the Unix time, in microseconds.
      It is updated in the server cron job and before executing a command.
      It is useful for complex call stacks, such as a command causing a
      key space notification, causing a module to execute a RedisModule_Call,
      causing another notification, etc.
      It makes sense that all these callbacks would use the same clock.
      45c99d70
  14. 26 Jul, 2022 3 commits
    • Huang Zhw's avatar
      When client tracking is on, invalidation message of flushdb in a (#11038) · 6f0a27e3
      Huang Zhw authored
      When FLUSHDB / FLUSHALL / SWAPDB is inside MULTI / EXEC, the
      client side tracking invalidation message was interleaved with transaction response.
      6f0a27e3
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #11030, use lua_rawget to avoid triggering metatables and crash. (#11032) · 020e046b
      Meir Shpilraien (Spielrein) authored
      Fix #11030, use lua_rawget to avoid triggering metatables.
      
      #11030 shows how return `_G` from the Lua script (either function or eval), cause the
      Lua interpreter to Panic and the Redis processes to exit with error code 1.
      Though return `_G` only panic on Redis 7 and 6.2.7, the underline issue exists on older
      versions as well (6.0 and 6.2). The underline issue is returning a table with a metatable
      such that the metatable raises an error.
      
      The following example demonstrate the issue:
      ```
      127.0.0.1:6379> eval "local a = {}; setmetatable(a,{__index=function() foo() end}) return a" 0
      Error: Server closed the connection
      ```
      ```
      PANIC: unprotected error in call to Lua API (user_script:1: Script attempted to access nonexistent global variable 'foo')
      ```
      
      The Lua panic happened because when returning the result to the client, Redis needs to
      introspect the returning table and transform the table into a resp. In order to scan the table,
      Redis uses `lua_gettable` api which might trigger the metatable (if exists) and might raise an error.
      This code is not running inside `pcall` (Lua protected call), so raising an error causes the
      Lua to panic and exit. Notice that this is not a crash, its a Lua panic that exit with error code 1.
      
      Returning `_G` panics on Redis 7 and 6.2.7 because on those versions `_G` has a metatable
      that raises error when trying to fetch a none existing key.
      
      ### Solution
      
      Instead of using `lua_gettable` that might raise error and cause the issue, use `lua_rawget`
      that simply return the value from the table without triggering any metatable logic.
      This is promised not to raise and error.
      
      The downside of this solution is that it might be considered as breaking change, if someone
      rely on metatable in the returned value. An alternative solution is to wrap this entire logic
      with `pcall` (Lua protected call), this alternative require a much bigger refactoring.
      
      ### Back Porting
      
      The same fix will work on older versions as well (6.2, 6.0). Notice that on those version,
      the issue can cause Redis to crash if inside the metatable logic there is an attempt to accesses
      Redis (`redis.call`). On 7.0, there is not crash and the `redis.call` is executed as if it was done
      from inside the script itself.
      
      ### Tests
      
      Tests was added the verify the fix
      020e046b
    • Viktor Söderqvist's avatar
      Gossip forgotten nodes on `CLUSTER FORGET` (#10869) · 5032de50
      Viktor Söderqvist authored
      Gossip the cluster node blacklist in ping and pong messages.
      This means that CLUSTER FORGET doesn't need to be sent to all nodes in a cluster.
      It can be sent to one or more nodes and then be propagated to the rest of them.
      
      For each blacklisted node, its node id and its remaining blacklist TTL is gossiped in a
      cluster bus ping extension (introduced in #9530).
      5032de50
  15. 19 Jul, 2022 1 commit
    • Binbin's avatar
      Fix timing issue in cluster test (#11008) · 5ce64ab0
      Binbin authored
      A timing issue like this was reported in freebsd daily CI:
      ```
      *** [err]: Sanity test push cmd after resharding in tests/unit/cluster/cli.tcl
      Expected 'CLUSTERDOWN The cluster is down' to match '*MOVED*'
      ```
      
      We additionally wait for each node to reach a consensus on the cluster
      state in wait_for_condition to avoid the cluster down error.
      
      The fix just like #10495, quoting madolson's comment:
      Cluster check just verifies the the config state is self-consistent,
      waiting for cluster_state to be okay is an independent check that all
      the nodes actually believe each other are healthy.
      
      At the same time i noticed that unit/moduleapi/cluster.tcl has an exact
      same test, may have the same problem, also modified it.
      5ce64ab0
  16. 18 Jul, 2022 2 commits
    • Oran Agra's avatar
      Fix heap overflow corruption in XAUTOCLAIM (CVE-2022-31144) (#11002) · 2825b605
      Oran Agra authored
      The temporary array for deleted entries reply of XAUTOCLAIM was
      insufficient, but also in fact the COUNT argument should be used to
      control the size of the reply, so instead of terminating the loop by
      only counting the claimed entries, we'll count deleted entries as well.
      
      Fix #10968
      Addresses CVE-2022-31144
      2825b605
    • ranshid's avatar
      Avoid using unsafe C functions (#10932) · eacca729
      ranshid authored
      replace use of:
      sprintf --> snprintf
      strcpy/strncpy  --> redis_strlcpy
      strcat/strncat  --> redis_strlcat
      
      **why are we making this change?**
      Much of the code uses some unsafe variants or deprecated buffer handling
      functions.
      While most cases are probably not presenting any issue on the known path
      programming errors and unterminated strings might lead to potential
      buffer overflows which are not covered by tests.
      
      **As part of this PR we change**
      1. added implementation for redis_strlcpy and redis_strlcat based on the strl implementation: https://linux.die.net/man/3/strl
      2. change all occurrences of use of sprintf with use of snprintf
      3. change occurrences of use of  strcpy/strncpy with redis_strlcpy
      4. change occurrences of use of strcat/strncat with redis_strlcat
      5. change the behavior of ll2string/ull2string/ld2string so that it will always place null
        termination ('\0') on the output buffer in the first index. this was done in order to make
        the use of these functions more safe in cases were the user will not check the output
        returned by them (for example in rdbRemoveTempFile)
      6. we added a compiler directive to issue a deprecation error in case a use of
        sprintf/strcpy/strcat is found during compilation which will result in error during compile time.
        However keep in mind that since the deprecation attribute is not supported on all compilers,
        this is expected to fail during push workflows.
      
      
      **NOTE:** while this is only an initial milestone. We might also consider
      using the *_s implementation provided by the C11 Extensions (however not
      yet widly supported). I would also suggest to start
      looking at static code analyzers to track unsafe use cases.
      For example LLVM clang checker supports security.insecureAPI.DeprecatedOrUnsafeBufferHandling
      which can help locate unsafe function usage.
      https://clang.llvm.org/docs/analyzer/checkers.html#security-insecureapi-deprecatedorunsafebufferhandling-c
      The main reason not to onboard it at this stage is that the alternative
      excepted by clang is to use the C11 extensions which are not always
      supported by stdlib.
      eacca729
  17. 17 Jul, 2022 2 commits
  18. 13 Jul, 2022 1 commit
    • Oran Agra's avatar
      Avoid valgrind fishy value warning on corrupt restore payloads (#10937) · 599e59eb
      Oran Agra authored
      The corrupt dump fuzzer uncovered a valgrind warning saying:
      ```
      ==76370== Argument 'size' of function malloc has a fishy (possibly negative) value: -3744781444216323815
      ```
      This allocation would have failed (returning NULL) and being handled properly by redis (even before this change), but we also want to silence the valgrind warnings (which are checking that casting to ssize_t produces a non-negative value).
      
      The solution i opted for is to explicitly fail these allocations (returning NULL), before even reaching `malloc` (which would have failed and return NULL too).
      
      The implication is that we will not be able to support a single allocation of more than 2GB on a 32bit system (which i don't think is a realistic scenario).
      i.e. i do think we could be facing cases were redis consumes more than 2gb on a 32bit system, but not in a single allocation.
      
      The byproduct of this, is that i dropped the overflow assertions, since these will now lead to the same OOM panic we have for failed allocations.
      599e59eb
  19. 12 Jul, 2022 1 commit
  20. 11 Jul, 2022 3 commits
    • Binbin's avatar
      Trying to fix cluster test (#10963) · 693acc01
      Binbin authored
      #10942 break the new test added in #10449
      ```
      Testing unit: 29-slot-migration-response.tcl
      Cluster Join and auto-discovery test: FAILED: Cluster failed to join into a full mesh.
      ```
      
      It looks like we need to wait for the cluster in 28 to become stable.
      693acc01
    • Binbin's avatar
      Add cluster-port support to redis-cli --cluster (#10344) · 35e8ae3e
      Binbin authored
      
      
      In #9389, we add a new `cluster-port` config and make cluster bus port configurable,
      and currently redis-cli --cluster create/add-node doesn't support with a configurable `cluster-port` instance.
      Because redis-cli uses the old way (port + 10000) to send the `CLUSTER MEET` command.
      
      Now we add this support on redis-cli `--cluster`, note we don't need to explicitly pass in the
      `cluster-port` parameter, we can get the real `cluster-port` of the node in `clusterManagerNodeLoadInfo`,
      so the `--cluster create` and `--cluster add-node` interfaces have not changed.
      
      We will use the `cluster-port` when we are doing `CLUSTER MEET`, also note that `CLUSTER MEET` bus-port
      parameter was added in 4.0, so if the bus_port (the one in redis-cli) is 0, or equal (port + 10000),
      we just call `CLUSTER MEET` with 2 arguments, using the old form.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      35e8ae3e
    • Madelyn Olson's avatar
      Fix crash during handshake and cluster shards call (#10942) · e6a1b2ea
      Madelyn Olson authored
      * Fix an engine crash when there are nodes in handshaking and a user calls cluster shards
      e6a1b2ea
  21. 04 Jul, 2022 2 commits
  22. 03 Jul, 2022 1 commit
  23. 30 Jun, 2022 1 commit
    • Binbin's avatar
      Add SENTINEL command flag to CLIENT/COMMANDS subcommands (#10904) · 35e836c2
      Binbin authored
      This was harmless because we marked the parent command
      with SENTINEL flag. So the populateCommandTable was ok.
      And we also don't show the flag (SENTINEL and ONLY-SENTNEL)
      in COMMAND INFO.
      
      In this PR, we also add the same CMD_SENTINEL and CMD_ONLY_SENTINEL
      flags check when populating the sub-commands.
      so that in the future it'll be possible to add some sub-commands to sentinel or sentinel-only but not others.
      35e836c2
  24. 29 Jun, 2022 1 commit
  25. 28 Jun, 2022 1 commit
    • jonnyomerredis's avatar
      Add sharded pubsub keychannel count to client info (#10895) · 35c2ee87
      jonnyomerredis authored
      When calling CLIENT INFO/LIST, and in various debug prints, Redis is printing
      the number of pubsub channels / patterns the client is subscribed to.
      With the addition of sharded pubsub, it would be useful to print the number of
      keychannels the client is subscribed to as well.
      35c2ee87
  26. 27 Jun, 2022 1 commit
    • Viktor Söderqvist's avatar
      Add missing REDISMODULE_CLIENTINFO_INITIALIZER (#10885) · 6af02100
      Viktor Söderqvist authored
      The module API docs mentions this macro, but it was not defined (so no one could have used it).
      
      Instead of adding it as is, we decided to add a _V1 macro, so that if / when we some day extend this struct,
      modules that use this API and don't need the extra fields, will still use the old version
      and still be compatible with older redis version (despite being compiled with newer redismodule.h)
      6af02100