1. 05 Apr, 2022 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Functions: Move library meta data to be part of the library payload. (#10500) · ae020e3d
      Meir Shpilraien (Spielrein) authored
      ## Move library meta data to be part of the library payload.
      
      Following the discussion on https://github.com/redis/redis/issues/10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.**
      
      The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example:
      
      ```lua
      #!lua name=test
      redis.register_function('foo', function() return 1 end)
      ```
      
      The above script will run on the Lua engine and will create a library called `test`.
      
      ## API Changes (compare to 7.0 rc2)
      
      * `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`.
      * The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST`
      
      
      ## Breaking Changes (compare to 7.0 rc2)
      
      * Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line).
      * Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid.
      
      ## Notes
      
      * Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).
      ae020e3d
    • bugwz's avatar
      Cluster node name sanity check (#10391) · 2db0d898
      bugwz authored
      
      
      * Limit cluster node id length for CLUSTER commands loading
      * Cluster node name sanity check for length and values
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      2db0d898
    • judeng's avatar
      delete obsolete REDISMODULE_EXPERIMENTAL_API define in module demos (#10527) · 9578b67e
      judeng authored
      This macro was recently removed from redismodule.h, so no longer needed.
      9578b67e
    • Binbin's avatar
      Remove outdated utils/hashtable/rehashing.c (#10526) · a95ae56b
      Binbin authored
      It is really outdated and i guessing no one will use it now. 
      Remove the corresponding utils/hashtable directory.
      closes #10235
      a95ae56b
  2. 04 Apr, 2022 4 commits
    • chenyang8094's avatar
      Fix null pointer subtraction warning (#10498) · cb625844
      chenyang8094 authored
      The warning:
      ```
      pqsort.c:106:7: warning: performing pointer subtraction with a null pointer has undefined behavior [-Wnull-pointer-subtraction]
      loop:   SWAPINIT(a, es);
              ^~~~~~~~~~~~~~~
      pqsort.c:65:47: note: expanded from macro 'SWAPINIT'
      #define SWAPINIT(a, es) swaptype = ((char *)a - (char *)NULL) % sizeof(long) || \
      ```
      Clang version:
      ```
      Apple clang version 13.1.6 (clang-1316.0.21.2)
      Target: x86_64-apple-darwin21.3.0
      Thread model: posix
      InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
      ```
      cb625844
    • Bofang Liu's avatar
      Fix mistake / outdated doc comment (#10521) · 8500ca4a
      Bofang Liu authored
      Unlike original reference code, Redis uses 64bit variable, so half the bits make it 32bit, not 16
      8500ca4a
    • Meir Shpilraien (Spielrein)'s avatar
      Fix #10508, on error, pop function and error handler from Lua stack. (#10519) · 047b6093
      Meir Shpilraien (Spielrein) authored
      If, for some reason, Redis decides not to execute the script, we need
      to pop the function and error handler from Lua stack. Otherwise, eventually
      the Lua stack will explode.
      
      Relevant only for 7.0-rc1 and 7.0-rc2.
      047b6093
    • judeng's avatar
      fix typos in aof.c (#10513) · 6cdaa277
      judeng authored
      
      
      function aofRewriteLimited in aof.c, deley->delay and NAX->MAX
      Co-authored-by: default avatarjudeng <judeng@didiglobal.com>
      6cdaa277
  3. 03 Apr, 2022 1 commit
  4. 02 Apr, 2022 1 commit
    • Viktor Söderqvist's avatar
      Turn into replica on SETSLOT (#10489) · b53c7f2c
      Viktor Söderqvist authored
      * Fix race condition where node loses its last slot and turns into replica
      
      When a node has lost its last slot and finds out from the SETSLOT command
      before the cluster bus PONG from the new owner arrives. In this case, the
      node didn't turn itself into a replica of the new slot owner.
      
      This commit adds the same logic to the SETSLOT command as already exists
      for the cluster bus PONG processing.
      
      * Revert "Fix new / failing cluster slot migration test (#10482)"
      
      This reverts commit 0b21ef8d.
      
      In this test, the old slot owner finds out that it has lost its last
      slot in a nondeterministic way. Either the cluster bus PONG from the
      new slot owner and sometimes in a SETSLOT command from redis-cli. In
      both cases, the result should be the same and the old owner should
      turn itself into a replica of the new slot owner.
      b53c7f2c
  5. 31 Mar, 2022 3 commits
    • sundb's avatar
      Fix failing moduleconfigs tests and memory leak (#10501) · b8eb2a73
      sundb authored
      Fix global `strval` not reset to NULL after being freed, causing a crash on alpine
      (most likely because the dynamic library loader doesn't init globals on reload)
      By the way, fix the memory leak of using `RedisModule_Free` to free `RedisModuleString`,
      and add a corresponding test.
      b8eb2a73
    • Madelyn Olson's avatar
      Prevent replica failover during manual takeover test (#10499) · e81bd15e
      Madelyn Olson authored
      During 11-manual-takeover.tcl, if the killing of the instances happens
      too slowly, one of the replicas might be able to promote itself.
      I'm not sure why it was slow, but it was observed taking 6 seconds
      which is enough time to do an election.
      I was able to verify the error locally by adding a small delay (1 second)
      during ASAN CI. A fix is just to disable automated failover until all the
      nodes are confirmed dead.
      e81bd15e
    • Binbin's avatar
      Fix cluster slot migration test (#10495) · a3075ca4
      Binbin authored
      Fix three timing issues in the test
      a3075ca4
  6. 30 Mar, 2022 6 commits
    • Ozan Tezcan's avatar
      Use exit code 1 on error in redis-cli (#10468) · 7da1cc3e
      Ozan Tezcan authored
      On error, redis-cli was returning `REDIS_ERR` on some cases by mistake. `REDIS_ERR` is `-1` which becomes `255` as exit code. This commit changes it and returns `1` on errors to be consistent.
      7da1cc3e
    • Binbin's avatar
      Move restart_killed_instances and verify_sentinel_auto_discovery to utils (#10497) · 6075f506
      Binbin authored
      Create a utils.tcl in sentinel/tests/includes, and move two procs to it.
      Allow sentinel test 08-hostname-conf run on its own.
      6075f506
    • dependabot[bot]'s avatar
      Bump actions/checkout from 2 to 3 (#10390) · 4e55d557
      dependabot[bot] authored
      Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
      - [Release notes](https://github.com/actions/checkout/releases)
      - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
      - [Commits](https://github.com/actions/checkout/compare/v2...v3
      
      )
      
      ---
      updated-dependencies:
      - dependency-name: actions/checkout
        dependency-type: direct:production
        update-type: version-update:semver-major
      ...
      Signed-off-by: default avatardependabot[bot] <support@github.com>
      Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      4e55d557
    • dependabot[bot]'s avatar
      Bump actions/cache from 2 to 3 (#10463) · 8df37363
      dependabot[bot] authored
      Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
      - [Release notes](https://github.com/actions/cache/releases)
      - [Commits](https://github.com/actions/cache/compare/v2...v3
      
      )
      
      ---
      updated-dependencies:
      - dependency-name: actions/cache
        dependency-type: direct:production
        update-type: version-update:semver-major
      ...
      Signed-off-by: default avatardependabot[bot] <support@github.com>
      Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      8df37363
    • Nick Chun's avatar
      Module Configurations (#10285) · bda9d74d
      Nick Chun authored
      This feature adds the ability to add four different types (Bool, Numeric,
      String, Enum) of configurations to a module to be accessed via the redis
      config file, and the CONFIG command.
      
      **Configuration Names**:
      
      We impose a restriction that a module configuration always starts with the
      module name and contains a '.' followed by the config name. If a module passes
      "config1" as the name to a register function, it will be registered as MODULENAME.config1.
      
      **Configuration Persistence**:
      
      Module Configurations exist only as long as a module is loaded. If a module is
      unloaded, the configurations are removed.
      There is now also a minimal core API for removal of standardConfig objects
      from configs by name.
      
      **Get and Set Callbacks**:
      
      Storage of config values is owned by the module that registers them, and provides
      callbacks for Redis to access and manipulate the values.
      This is exposed through a GET and SET callback.
      
      The get callback returns a typed value of ...
      bda9d74d
    • Binbin's avatar
      command json files cleanups (#10473) · e2fa6aa1
      Binbin authored
      This PR do some command json files cleanups:
      
      1. Add COMMAND TIPS to some commands
      - command-docs: add `NONDETERMINISTIC_OUTPUT_ORDER`
      - command-info: add `NONDETERMINISTIC_OUTPUT_ORDER`
      - command-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
      - command: change `NONDETERMINISTIC_OUTPUT` to `NONDETERMINISTIC_OUTPUT_ORDER`
      - function-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
      - latency-doctor: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL`
      - latency-graph: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL`
      - memory-doctor: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL`
      - memory-malloc-stats: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL`
      - memory-purge: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:ALL_SUCCEEDED`
      - module-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
      - msetnx: add `REQUEST_POLICY:MULTI_SHARD` and `RESPONSE_POLICY:AGG_MIN`
      - object-refcount: add `NONDETERMINISTIC_OUTPUT`
      3. Only (mostly) indentation and formatting changes:
      - cluster-shards
      - latency-history
      - pubsub-shardchannels
      - pubsub-shardnumsub
      - spublish
      - ssubscribe
      - sunsubscribe
      4. add doc_flags (DEPRECATED) to cluster-slots,  replaced_by `CLUSTER SHARDS` in 7.0
      5. command-getkeysandflags: a better summary (the old one is copy from command-getkeys)
      6. adjustment of command parameter types
      - `port` is integer, not string (`MIGRATE`, `REPLICAOF`, `SLAVEOF`)
      - `replicationid` is string, not integer (`PSYNC`)
      - `pattern` is pattern, not string (`PUBSUB CHANNELS`, `SENTINEL RESET`, `SORT`, `SORT_RO`)
      e2fa6aa1
  7. 29 Mar, 2022 4 commits
    • Oran Agra's avatar
      fix daily.yaml skip filters (#10490) · 16d206ee
      Oran Agra authored
      * missing parenthesis meant that the ubuntu and centos jobs were not
        skipped
      * the recently divided freebsd, macos, and valgrind jobs, which are now
        split into distict jobs for redis, modules, sentinel, cluster. were
        all executed, producing a build, but not running anything.
        now they're filtered at the job level
      * iothreads was missing from the skip list defaults, so was not skipped
      16d206ee
    • Viktor Söderqvist's avatar
      redis-cli: Do DNS lookup before sending CLUSTER MEET (#10436) · 35bb0212
      Viktor Söderqvist authored
      Affects `--cluster create` and `--cluster add-node`.
      35bb0212
    • zhaozhao.zz's avatar
      show cluster.links in MEMORY STATS (#10302) · 001e1925
      zhaozhao.zz authored
      001e1925
    • Oran Agra's avatar
      improve malloc efficiency for cluster slots_info_pairs (#10488) · 3b1e65a3
      Oran Agra authored
      This commit improve malloc efficiency of the slots_info_pairs mechanism in cluster.c
      by changing adlist into an array being realloced with greedy growth mechanism
      
      Recently the cluster tests are consistently failing when executed with ASAN in the CI.
      I tried to track down the commit that started it, and it appears to be #10293.
      Looking at the commit, i realize it didn't affect this test / flow, other than the
      replacement of the slots_info_pairs from sds to list.
      
      I concluded that what could be happening is that the slot range is very fragmented,
      and that results in many allocations.
      with sds, it results in one allocation and also, we have a greedy growth mechanism,
      but with adlist, we just have many many small allocations.
      this probably causes stress on ASAN, and causes it to be slow at termination.
      3b1e65a3
  8. 28 Mar, 2022 4 commits
  9. 27 Mar, 2022 3 commits
    • Oran Agra's avatar
      Fix new / failing cluster slot migration test (#10482) · 0b21ef8d
      Oran Agra authored
      #10381 fixed an issue in `redis-cli --cluster reshard` that used to fail it (redis-cli) because
      of a race condition.
      the race condition is / was that when moving the last slot from a node, sometimes the PONG
      messages delivering the configuration change arrive to that node before the SETSLOT arrives
      to it, and it becomes a replica.
      other times the the SETSLOT arrive first, and then PONG **doesn't** demote it.
      
      **however**, the PR also added a new test that suffers from exactly the same race condition,
      and the tests started failing a lot.
      
      The fact is (if i understand it correctly), that this test (the one being deleted here), isn't related
      to the fix that PR fixed (which was to fix redis-cli).
      The race condition in the cluster code still happens, and as long as we don't solve it, there's
      no reason to test it.
      
      For now, even if my understandings are wrong, i'm gonna delete that failing test, since as far as
      i understand, #10381 didn't introduce any new risks for that matter (which are gonna be
      compromised by removing this check), this race existed since forever, and still exists, and the
      fact that redis-cli is now immune to it is still being tested.
      
      Additional work should be carried to fix it, and i live it for other PRs to handle.
      0b21ef8d
    • Moti Cohen's avatar
      Fix Sentinel reconnect test following ACL change (#10480) · 37d761ba
      Moti Cohen authored
      Replace condition with wait_for_condition On "Verify sentinel that restarted 
      failed to reconnect master after ACL change"
      
      The reason we reach it, is because the test is fast enough to modify ACL and
      test sentinel connection status with the server - before its scheduled operation
      got the chance to update connection status with the server:
      ```
      /* Perform scheduled operations for the specified Redis instance. */
      void sentinelHandleRedisInstance(sentinelRedisInstance *ri) {
          /* ========== MONITORING HALF ============ */
          /* Every kind of instance */
          sentinelReconnectInstance(ri);
      ```
      37d761ba
    • Binbin's avatar
      Cleanups in redis.conf (#10452) · f25e688e
      Binbin authored
      Did some cleanups:
      1. local local typo
      2. replace the only slave word in the file
      3. add FUNCTION FLUSH to `lazyfree-lazy-user-flush` description
      4. thought it would be better to use these, there are actually "four" options
      5. the the typo
      6. remove a extra space
      7. change comment next to `activedefrag no` to match the default value
      f25e688e
  10. 26 Mar, 2022 1 commit
  11. 25 Mar, 2022 1 commit
    • zhaozhao.zz's avatar
      optimize(remove) usage of client's pending_querybuf (#10413) · 78bef6e1
      zhaozhao.zz authored
      To remove `pending_querybuf`, the key point is reusing `querybuf`, it means master client's `querybuf` is not only used to parse command, but also proxy to sub-replicas.
      
      1. add a new variable `repl_applied` for master client to record how many data applied (propagated via `replicationFeedStreamFromMasterStream()`) but not trimmed in `querybuf`.
      
      2. don't sdsrange `querybuf` in `commandProcessed()`, we trim it to `repl_applied` after the whole replication pipeline processed to avoid fragmented `sdsrange`. And here are some scenarios we cannot trim to `qb_pos`:
          * we don't receive complete command from master
          * master client blocked because of client pause
          * IO threads operate read, master client flagged with CLIENT_PENDING_COMMAND
      
          In these scenarios, `qb_pos` points to the part of the current command or the beginning of next command, and the current command is not applied yet, so the `repl_applied` is not equal to `qb_pos`.
      
      Some other notes:
      * Do not do big arg optimization on master client, since we can only sdsrange `querybuf` after data sent to replicas.
      * Set `qb_pos` and `repl_applied` to 0 when `freeClient` in `replicationCacheMaster`.
      * Rewrite `processPendingCommandsAndResetClient` to `processPendingCommandAndInputBuffer`, let `processInputBuffer` to be called successively after `processCommandAndResetClient`.
      78bef6e1
  12. 22 Mar, 2022 5 commits
    • Oran Agra's avatar
      Split daily CI into smaller chunks (#10469) · 1a57af62
      Oran Agra authored
      this should aid find the CI issues with freebsd and macos runs, and also
      get faster results from valgrind and tls
      1a57af62
    • Oran Agra's avatar
      crash log, print killer pid only when si_code is SI_USER (#10454) · 6761d10c
      Oran Agra authored
      Avoid printing "Killed by PID" when si_code != SI_USER.
      Apparently SI_USER isn't always set to 0. e.g. on Mac it's 0x10001 and the check that did <= was wrong.
      6761d10c
    • Meir Shpilraien (Spielrein)'s avatar
      Add new RM_Call flags for script mode, no writes, and error replies. (#10372) · f3855a09
      Meir Shpilraien (Spielrein) authored
      The PR extends RM_Call with 3 new capabilities using new flags that
      are given to RM_Call as part of the `fmt` argument.
      It aims to assist modules that are getting a list of commands to be
      executed from the user (not hard coded as part of the module logic),
      think of a module that implements a new scripting language...
      
      * `S` - Run the command in a script mode, this means that it will raise an
        error if a command which are not allowed inside a script (flaged with the
        `deny-script` flag) is invoked (like SHUTDOWN). In addition, on script mode,
        write commands are not allowed if there is not enough good replicas (as
        configured with `min-replicas-to-write`) and/or a disk error happened.
      
      * `W` - no writes mode, Redis will reject any command that is marked with `write`
        flag. Again can be useful to modules that implement a new scripting language
        and wants to prevent any write commands.
      
      * `E` - Return errors as RedisModuleCallReply. Today the errors that happened
        before the command was invoked (like unknown commands or acl error) return
        a NULL reply and set errno. This might be missing important information about
        the failure and it is also impossible to just pass the error to the user using
        RM_ReplyWithCallReply. This new flag allows you to get a RedisModuleCallReply
        object with the relevant error message and treat it as if it was an error that was
        raised by the command invocation.
      
      Tests were added to verify the new code paths.
      
      In addition small refactoring was done to share some code between modules,
      scripts, and `processCommand` function:
      1. `getAclErrorMessage` was added to `acl.c` to unified to log message extraction
        from the acl result
      2. `checkGoodReplicasStatus` was added to `replication.c` to check the status of
        good replicas. It is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and
        `processCommand`.
      3. `writeCommandsGetDiskErrorMessage` was added to `server.c` to get the error
        message on persistence failure. Again it is used on `scriptVerifyWriteCommandAllow`,
        `RM_Call`, and `processCommand`.
      f3855a09
    • yiyuaner's avatar
      Fix an off by one error in zzlStrtod (#10465) · 08aed7e7
      yiyuaner authored
      When vlen = sizeof(buf), the statement buf[vlen] = '\0' accessing the buffer buf is an off by one error.
      08aed7e7
    • zhaozhao.zz's avatar
      config rewrite enhancement, in case of line longer than 1024 (#8009) · 79db037a
      zhaozhao.zz authored
      When rewrite the config file, we need read the old config file first,
      but the CONFIG_MAX_LEN is 1024, so if some lines are longer than it,
      it will generate a wrong config file, and redis cannot reboot from
      the new config file.
      
      Rename CONFIG_MAX_LINE to CONFIG_READ_LEN
      79db037a
  13. 21 Mar, 2022 3 commits