1. 13 Jun, 2023 2 commits
    • Harkrishn Patro's avatar
      Allow cluster slots/shards api to respond during loading (#12269) · a9e32767
      Harkrishn Patro authored
      It would be helpful for clients to get cluster slots/shards information during a node failover and is loading data.
      a9e32767
    • Binbin's avatar
      Fix XREADGROUP BLOCK stuck in endless loop (#12301) · e7129e43
      Binbin authored
      
      
      For the XREADGROUP BLOCK > scenario, there is an endless loop.
      Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command
      
      The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
      looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
      and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
      i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
      is problematic since the code keeps accessing the global list.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      e7129e43
  2. 12 Jun, 2023 1 commit
  3. 11 Jun, 2023 4 commits
    • YaacovHazan's avatar
      Reset command duration for rejected command. (#12247) · 0bfb6d55
      YaacovHazan authored
      In 7.2, After 971b177f
      
       we make sure (assert) that
      the duration has been recorded when resetting the client.
      
      This is not true for rejected commands.
      The use case I found is a blocking command that an ACL rule changed before
      it was unblocked, and while reprocessing it, the command rejected and triggered the assert.
      
      The PR reset the command duration inside rejectCommand / rejectCommandSds.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0bfb6d55
    • Chen Tianjie's avatar
      Allow bigger tolerance in eventloop duration test. (#12179) · 2d7d3911
      Chen Tianjie authored
      In #11963, some new tests about eventloop duration were added, which includes time measurement in TCL scripts. This has caused some unexpected CI failures, such as #12169 and #12177, due to slow test servers or some performance jittering.
      2d7d3911
    • Wen Hui's avatar
      Adding missing test cases for Addslot Command (#12288) · d412269f
      Wen Hui authored
      Added missing test case coverage for below scenarios:
      
      1. The command only works if all the specified slots are, from
        the point of view of the node receiving the command, currently
        not assigned. A node will refuse to take ownership for slots that
        already belong to some other node (including itself).
      2. The command fails if the same slot is specified multiple times.
      d412269f
    • Binbin's avatar
      Fix ACLAppendUserForLoading memory leak when merge error (#12296) · 5fd9756d
      Binbin authored
      This leak will only happen in loadServerConfigFromString,
      that is, when we are loading a redis.conf, and the user is wrong.
      
      Because it happens in loadServerConfigFromString, redis will
      exit if there is an error, so this is actually just a cleanup.
      5fd9756d
  4. 08 Jun, 2023 1 commit
  5. 06 Jun, 2023 1 commit
  6. 05 Jun, 2023 1 commit
  7. 04 Jun, 2023 1 commit
    • Yossi Gottlieb's avatar
      Update deps/README.md. · 032bb2a2
      Yossi Gottlieb authored
      * We patch hiredis rather than rely on having a compatible sds version.
      * We now have better test coverage for redis-cli and redis-benchmark.
      032bb2a2
  8. 30 May, 2023 3 commits
    • Yossi Gottlieb's avatar
      2eaf2360
    • Yossi Gottlieb's avatar
      Squashed 'deps/hiredis/' changes from f8de9a4bd..b6a052fe0 · fad6c713
      Yossi Gottlieb authored
      b6a052fe0 Helper for setting TCP_USER_TIMEOUT socket option (#1188)
      3fa9b6944 Add RedisModule adapter (#1182)
      d13c091e9 Fix wincrypt symbols conflict
      5d84c8cfd Add a test ensuring we don't clobber connection error.
      3f95fcdae Don't attempt to set a timeout if we are in an error state.
      aacb84b8d Fix typo in makefile.
      563b062e3 Accept -nan per the RESP3 spec recommendation.
      04c1b5b02 Fix colliding option values
      4ca8e73f6 Rework searching for openssl
      cd208812f Attempt to find the correct path for openssl.
      011f7093c Allow specifying the keepalive interval
      e9243d4f7 Cmake static or shared (#1160)
      1cbd5bc76 Write a version file for the CMake package (#1165)
      6f5bae8c6 fix typo
      acd09461d CMakeLists.txt: respect BUILD_SHARED_LIBS
      97fcf0fd1 Add sdevent adapter
      ccff093bc Bump dev version for the next release cycle.
      c14775b4e Prepare for v1.1.0 GA
      f0bdf8405 Add support for nan in RESP3 double (#1133)
      991b0b0b3 Add an example that calls redisCommandArgv (#1140)
      a36686f84 CI updates (#1139)
      8ad4985e9 fix flag reference
      7583ebb1b Make freeing a NULL redisAsyncContext a no op.
      2c53dea7f Update version in dev branch.
      f063370ed Prepare for v1.1.0-rc1
      2b069573a CI fixes in preparation of release
      e1e9eb40d Add author information to release-drafter template.
      afc29ee1a Update for mingw cross compile
      ceb8a8815 fixed cpp build error with adapters/libhv.h
      3b15a04b5 Fixup of PR734: Coverage of hiredis.c (#1124)
      c245df9fb CMake corrections for building on Windows (#1122)
      9c338a598 Fix PUSH handler tests for Redis >= 7.0.5
      6d5c3ee74 Install on windows fixes (#1117)
      68b29e1ad Add timeout support to libhv adapter. (#1109)
      722e3409c Additional include directory given by pkg-config (#1118)
      bd9ccb8c4 Use __attribute__ when building with clang on windows
      5392adc26 set default SSL certificate directory
      560e66486 Minor refactor
      d756f68a5 Add libhv example to our standard Makefile
      a66916719 Add adapters/libhv
      855b48a81 Fix pkgconfig for hiredis_ssl
      79ae5ffc6 Fix protocol error (#1106)
      61b5b299f Use a windows specific keepalive function. (#1104)
      fce8abc1c Introduce .close method for redisContextFuncs
      cfb6ca881 Add REDIS_OPT_PREFER_UNSPEC (#1101)
      cc7c35ce6 Update documentation to explain redisConnectWithOptions.
      bc8d837b7 fix heap-buffer-overflow (#957)
      ca4a0e850 uvadapter: reduce number of uv_poll_start calls
      35d398c90 Fix cmake config path on Linux. CMake config files were installed to `/usr/local/share/hiredis`, which is not recognizable by `find_package()`. I'm not sure why it was set that way. Given the commit introducing it is for Windows, I keep that behavior consistent there, but fix the rest.
      10c78c6e1 Add possibility to prefer IPv6, IPv4 or unspecified
      1abe0c828 fuzzer: No alloc in redisFormatCommand() when fail
      329eaf9ba Fix heap-buffer-overflow issue in redisvFormatCommad
      eaae7321c Polling adapter requires sockcompat.h
      0a5fa3dde Regression test for off-by-one parsing error
      9e174e8f7 Add do while(0) protection for macros
      4ad99c69a Rework asSleep to be a generic millisleep function.
      75cb6c1ea Do store command timeout in the context for redisSetTimeout (#593)
      c57cad658 CMake: remove dict.c form hiredis_sources
      8491a65a9 Add Github Actions CI workflow for hiredis: Arm, Arm64, 386, windows. (#943)
      77e4f09ea Merge pull request #964 from afcidk/fix-createDoubleObject
      9219f7e7c Merge pull request #901 from devnexen/illumos_test_fix
      810cc6104 Merge pull request #905 from sundb/master
      df8b74d69 Merge pull request #1091 from redis/ssl-error-ub-fix
      0ed6cdec3 Fix some undefined behaviour
      507a6dcaa Merge pull request #1090 from Nordix/subscribe-oom-error
      b044eaa6a Copy error to redisAsyncContext when finding subscribe cb
      e0200b797 Merge pull request #1087 from redis/const-and-non-const-callback
      6a3e96ad2 Maintain backward compatibiliy withour onConnect callback.
      e7afd998f Merge pull request #1079 from SukkaW/drop-macos-10.15-runner
      17c8fe079 Merge pull request #931 from kristjanvalur/pr2
      b808c0c20 Merge pull request #1083 from chayim/ck-drafter
      367a82bf0 Merge pull request #1085 from stanhu/ssl-improve-options-setting
      71119a71d Make it possible to set SSL verify mode
      dd7979ac1 Merge pull request #1084 from stanhu/sh-improve-ssl-docs
      c71116178 Improve example for SSL initialization in README.md
      5c9b6b571 Release drafter
      a606ccf2a CI: use recommended `vmactions/freebsd-vm@v0`
      0865c115b Merge pull request #1080 from Nordix/readme-corrections
      f6cee7142 Fix README typos
      06be7ff31 Merge pull request #1050 from smmir-cent/fix-cmake-version
      7dd833d54 CI: bump macos runner version
      f69fac769 Drop `const` on redisAsyncContext in redisConnectCallback Since the callback is now re-entrant, it can call apis such as redisAsyncDisconnect()
      005d7edeb Support calling redisAsyncDisconnect from the onConnected callback, by deferring context deletion
      6ed060920 Add async regression test for issue #931
      eaa2a7ee7 Merge pull request #932 from kristjanvalur/pr3
      2ccef30f3 Add regression test for issue #945
      4b901d44a Initial async tests
      31c91408e Polling adapter and example
      8a15f4d65 Merge pull request #1057 from orgads/static-name
      902dd047f Merge pull request #1054 from kristjanvalur/pr08
      c78d0926b Merge pull request #1074 from michael-grunder/kristjanvalur-pr4
      2b115d56c Whitespace
      1343988ce Fix typos
      47b57aa24 Add some documentation on connect/disconnect callbacks and command callbacks
      a890d9ce2 Merge pull request #1073 from michael-grunder/kristjanvalur-pr1
      f246ee433 Whitespace, style
      94c1985bd Use correct type for getsockopt()
      5e002bc21 Support failed async connects on windows.
      5d68ad2f4 Merge pull request #1072 from michael-grunder/fix-redis7-unit-tests
      f4b6ed289 Fix tests so they work for Redis 7.0
      95a0c1283 Merge pull request #1058 from orgads/win64
      eedb37a65 Fix warnings on Win64
      47c3ecefc Merge pull request #1062 from yossigo/fix-push-notification-order
      e23d91c97 Merge pull request #1061 from yossigo/update-redis-apt
      34211ad54 Merge pull request #1063 from redis/fix-windows-tests
      9957af7e3 Whitelist hiredis repo path in cygwin
      b455b3381 Handle push notifications before or after reply.
      aed9ce446 Use official repository for redis package.
      d7683f35a Merge pull request #1047 from Nordix/unsubscribe-handling
      7c44a9d7e Merge pull request #1045 from Nordix/sds-updates
      dd4bf9783 Use the same name for static and shared libraries
      ff57c18b9 Embed debug information in windows static lib, rather than create a .pdb file
      8310ad4f5 fix cmake version
      7123b87f6 Handle any pipelined unsubscribe in async
      b6fb548fc Ignore pubsub replies without a channel/pattern
      00b82683b Handle overflows as errors instead of asserting
      64062a1d4 Catch size_t overflows in sds.c
      066c6de79 Use size_t/long to avoid truncation
      c6657ef65 Merge branch 'redis:master' into master
      50cdcab49 Fix potential fault at createDoubleObject
      fd033e983 Remove semicolon after do-while in _EL_CLEANUP
      664c415e7 Illumos test fixes, error message difference fot bad hostname test.
      
      git-subtree-dir: deps/hiredis
      git-subtree-split: b6a052fe0959dae69e16b9d74449faeb1b70dbe1
      fad6c713
    • Oran Agra's avatar
      Avoid trying to trim string loaded from RDB file. (#12241) · c2f1815b
      Oran Agra authored
      This is a followup fix for #11817
      c2f1815b
  9. 29 May, 2023 3 commits
    • Binbin's avatar
      Fix GETEX db delete call to emit DB_FLAG_KEY_EXPIRED on expiration (#12243) · 0dfb0250
      Binbin authored
      We should emit DB_FLAG_KEY_EXPIRED instead of DB_FLAG_KEY_DELETED.
      This is an overlook in #9406.
      0dfb0250
    • Binbin's avatar
      Try to fix SENTINEL SIMULATE-FAILURE test by re-source init-tests before each test (#12194) · da8f7428
      Binbin authored
      This test was introduced in #12079, it works well most of the time, but
      occasionally fails:
      ```
      00:34:45> SENTINEL SIMULATE-FAILURE crash-after-election works: OK
      00:34:45> SENTINEL SIMULATE-FAILURE crash-after-promotion works: FAILED: Sentinel set crash-after-promotion but did not exit
      ```
      
      Don't know the reason, it may be affected by the exit of the previous
      crash-after-election test. Because it doesn't really make much sense to
      go deeper into it now, we re-source init-tests to get a clean environment
      before each test, to try to fix this.
      
      After applying this change, we found a new error:
      ```
      16:39:33> SENTINEL SIMULATE-FAILURE crash-after-election works: FAILED: caught an error in the test couldn't open socket: connection refused
      couldn't open socket: connection refused
      ```
      
      I am guessing the sentinel triggers failover and exits before SENTINEL FAILOVER,
      added a new || condition in wait_for_condition to fix it.
      da8f7428
    • Binbin's avatar
      Try lazyfree temp zset in ZUNION / ZINTER / ZDIFF and optimize ZINTERCARD to... · 32f45215
      Binbin authored
      Try lazyfree temp zset in ZUNION / ZINTER / ZDIFF and optimize ZINTERCARD to avoid create temp zset (#12229)
      
      We check lazyfree_lazy_server_del in sunionDiffGenericCommand
      to see if we need to lazyfree the temp set. Now do the same in
      zunionInterDiffGenericCommand to lazyfree the temp zset.
      
      This is a minor change, follow #5903. Also improved the comments.
      
      Additionally, avoid creating unused zset object in ZINTERCARD,
      results in some 10% performance improvement. 
      32f45215
  10. 28 May, 2023 5 commits
  11. 26 May, 2023 1 commit
    • Binbin's avatar
      Add new loglevel nothing to sentinel-config.json (#12235) · e775b34e
      Binbin authored
      It was missing in #12223, and the reply-schemas daily
      was failing:
      ```
      jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas
      
      Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
          {'oneOf': [{'const': 'debug'},
                     {'const': 'verbose'},
                     {'const': 'notice'},
                     {'const': 'warning'},
                     {'const': 'unknown'}]}
      
      On instance['loglevel']:
          'nothing'
      ```
      e775b34e
  12. 24 May, 2023 7 commits
    • mojh7's avatar
      Fix typo functoin -> function (#12218) · 627c610f
      mojh7 authored
      functoin -> function
      627c610f
    • Oran Agra's avatar
      Make a light weight version (default) of DEBUG HTSTATS (#12212) · 3ca451c4
      Oran Agra authored
      The light version only shows the table sizes, while the pre-existing
      version that shows chain length stats is reachable with the `full` argument.
      
      This should allow looking into rehashing state, even on huge dicts, on
      which we're afraid to run the command for fear of causing a server freeze.
      
      Also, fix a possible overflow in dictGetStats.
      3ca451c4
    • Oran Agra's avatar
      CI to validate commands.def is up to date (#12227) · c871db24
      Oran Agra authored
      and update recent SENTINEL CONFIG changes.
      c871db24
    • judeng's avatar
      postpone the initialization of oject's lru&lfu until it is added to the db as... · d71478a8
      judeng authored
      postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
      
      This pr can get two performance benefits:
      1. Stop redundant initialization when most robj objects are created
      2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`
      
      Another code optimization:
      deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
      robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
      d71478a8
    • Wen Hui's avatar
      Adding test case for hvals, hkeys, hexists against wrong type (#12198) · d6648899
      Wen Hui authored
      HVALS, HKEYS and HEXISTS commands wrong type test cases were not covered so added the test cases.
      d6648899
    • Binbin's avatar
      Sync the new loglevel nothing to sentinel (#12223) · d0994c5b
      Binbin authored
      We add a new loglevel 'nothing' to disable logging in #12133.
      This PR syncs that config change to sentinel. Because in #11214
      we support modifying loglevel in runtime.
      
      Although I think sentinel doesn't need this nothing config,
      it's better to be consistent.
      d0994c5b
    • Binbin's avatar
      Add dummy CLUSTER SLAVES call tests to fix reply ci (#12224) · ec5721d6
      Binbin authored
      In #12166, we removed a call to CLUSTER SLAVES, which
      then caused reply-schemas ci to fail:
      ```
      WARNING! The following commands were not hit at all:
        cluster|slaves
        ERROR! at least one command was not hit by the tests
      ```
      
      Because we already have command output that cover CLUSTER REPLICAS
      elsewhere, here we simply add some dummy tests to fix the ci.
      ec5721d6
  13. 23 May, 2023 3 commits
    • Ping Xie's avatar
      Exclude aux fields from "cluster nodes" and "cluster replicas" output (#12166) · 4c74dd98
      Ping Xie authored
      This commit excludes aux fields from the output of the `cluster nodes` and `cluster replicas` command.
      We may decide to re-introduce them in some form or another in the future, but not in v7.2.
      4c74dd98
    • zhaozhao.zz's avatar
      add a new loglevel 'nothing' to disable logging (#12133) · 07ea2204
      zhaozhao.zz authored
      Users can record logs of different levels by setting the `loglevel`.
      However, sometimes there are many logs even at the warning level,
      which can affect the performance of Redis.
      
      For example, when a user accesses the tls-port using a non-encrypted link,
      Redis will log lots of "# Error accepting a client connection: ...".
      
      We can provide the ability to disable logging so that users can temporarily turn
      off logging and turn it back on after the problem is resolved.
      07ea2204
    • Shaya Potter's avatar
      Fix memory leak when RM_Call's RUN_AS_USER fails (#12158) · 71e6abe4
      Shaya Potter authored
      previously the argv wasn't freed so would leak.  not a common case, but should be handled.
      
      Solution: move RUN_AS_USER setup and error exit to the right place.
      this way, when we do `goto cleanup` (instead of return) it'll automatically do the right thing (including autoMemoryAdd)
      Removed the user argument from moduleAllocTempClient (reverted to the state before 6e993a5d
      
      )
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      71e6abe4
  14. 22 May, 2023 2 commits
    • Binbin's avatar
      Optimize HRANDFIELD and ZRANDMEMBER case 3 when listpack encoded (#12205) · 006ab26c
      Binbin authored
      Optimized HRANDFIELD and ZRANDMEMBER commands as in #8444,
      CASE 3 under listpack encoding. Boost optimization to CASE 2.5. 
      
      CASE 2.5 listpack only. Sampling unique elements, in non-random order.
      Listpack encoded hashes / zsets are meant to be relatively small, so
      HRANDFIELD_SUB_STRATEGY_MUL / ZRANDMEMBER_SUB_STRATEGY_MUL
      isn't necessary and we rather not make copies of the entries. Instead, we
      emit them directly to the output buffer.
      
      Simple benchmarks shows it provides some 400% improvement in HRANDFIELD
      and ZRANGESTORE both in CASE 3.
      
      Unrelated changes: remove useless setTypeRandomElements and fix a typo.
      006ab26c
    • binfeng-xin's avatar
      optimize spopwithcount propagation (#12082) · 38e284f1
      binfeng-xin authored
      
      
      A single SPOP with command with count argument resulted in many SPOP
      commands being propagated to the replica.
      This is inefficient because the key name is repeated many times, and is also
      being looked-up many times.
      also it results in high QPS metrics on the replica.
      To solve that, we flush batches of 1024 fields per SPOP command.
      Co-authored-by: default avatarzhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
      38e284f1
  15. 18 May, 2023 4 commits
    • Binbin's avatar
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and... · 48757934
      Binbin authored
      Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and expand dict in advance (#12185)
      
      For zsets that will eventually be stored as the skiplist encoding (has a dict),
      we can convert it to skiplist ahead of time. This change checks the number
      of arguments in the ZADD command, and converts the data-structure
      if the number of new entries exceeds the listpack-max-entries configuration.
      This can cause us to over-allocate memory if there are duplicate entries in the
      input, which is unexpected.
      
      For ZRANGESTORE, we know the size of the zset, so we can expand
      the dict in advance, to avoid the temporary dict from being rehashed
      while it grows.
      
      Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
      48757934
    • Hanna Fadida's avatar
      Add BITFIELD_RO basic tests for non-repl use cases (#12187) · 37cf1984
      Hanna Fadida authored
      Current tests for BITFIELD_RO command are skipped in the external mode,
      and therefore reply-schemas-validator reports a coverage error.
      This PR adds basic tests to increase coverage.
      37cf1984
    • Brennan's avatar
      Prevent repetitive backlog trimming (#12155) · 40e6131b
      Brennan authored
      When `replicationFeedSlaves()` serializes a command, it repeatedly calls
      `feedReplicationBuffer()` to feed it to the replication backlog piece by piece.
      It is unnecessary to call `incrementalTrimReplicationBacklog()` for every small
      amount of data added with `feedReplicationBuffer()` as the chance of the conditions
      being met for trimming are very low and these frequent calls add up to a notable
      performance cost. Instead, we will only attempt trimming when a new block is added
      to the replication backlog.
      
      Using redis-benchmark to saturate a local redis server indicated a performance
      improvement of around 3-3.5% for 100 byte SET commands with this change.
      40e6131b
    • YaacovHazan's avatar
      fix CMD_CALL_FROM_MODULE value (#12195) · 49845c24
      YaacovHazan authored
      CMD_CALL_FROM_MODULE overlapped CMD_CALL_REPROCESSING,
      changing its value to (1<<3)
      49845c24
  16. 17 May, 2023 1 commit