1. 18 Oct, 2023 4 commits
    • Oran Agra's avatar
      Redis 6.2.14 · 91863dd8
      Oran Agra authored
      91863dd8
    • Jachin's avatar
      Fix compile on macOS 13 (#12611) · 2f27ebb4
      Jachin authored
      Use the __MAC_OS_X_VERSION_MIN_REQUIRED macro to detect the
      macOS system version instead of using MAC_OS_X_VERSION_10_6.
      
      From MacOSX14.0.sdk, the default definitions of MAC_OS_X_VERSION_xxx have
      been removed in usr/include/AvailabilityMacros.h. It includes AvailabilityVersions.h,
      where the following condition must be met:
      `#if (!defined(_POSIX_C_SOURCE) && !defined(_XOPEN_SOURCE)) || defined(_DARWIN_C_SOURCE)`
      Only then will MAC_OS_X_VERSION_xxx be defined.
      However, in the project, _DARWIN_C_SOURCE is not defined, which leads to the
      loss of the definition for MAC_OS_X_VERSION_10_6.
      
      (cherry picked from commit a2b0701d)
      2f27ebb4
    • sundb's avatar
      Skip test for sdsRemoveFreeSpace when mem_allocator is not jemalloc (#11878) · 89391e42
      sundb authored
      Test `trim on SET with big value` (introduced from #11817) fails under mac m1 with libc mem_allocator.
      The reason is that malloc(33000) will allocate 65536 bytes(>42000).
      This test still passes under ubuntu with libc mem_allocator.
      
      ```
      *** [err]: trim on SET with big value in tests/unit/type/string.tcl
      Expected [r memory usage key] < 42000 (context: type source line 471 file /Users/iospack/data/redis_fork/tests/unit/type/string.tcl cmd {assert {[r memory usage key] < 42000}} proc ::test)
      ```
      
      simple test under mac m1 with libc mem_allocator:
      ```c
      void *p = zmalloc(33000);
      printf("malloc size: %zu\n", zmalloc_size(p));
      
      # output
      malloc size: 65536
      ```
      
      (cherry picked from commit 3fba3ccd)
      (cherry picked from commit 646069a9)
      89391e42
    • Yossi Gottlieb's avatar
      Fix issue of listen before chmod on Unix sockets (CVE-2023-45145) · 8f30d2a6
      Yossi Gottlieb authored
      Before this commit, Unix socket setup performed chmod(2) on the socket
      file after calling listen(2). Depending on what umask is used, this
      could leave the file with the wrong permissions for a short period of
      time. As a result, another process could exploit this race condition and
      establish a connection that would otherwise not be possible.
      
      We now make sure the socket permissions are set up prior to calling
      listen(2).
      
      (cherry picked from commit a11b3bc34a054818f2ac70e50adfc542ca1cba42)
      (cherry picked from commit 3ca02a0c541e019d77014424e207123160520cec)
      8f30d2a6
  2. 10 Jul, 2023 4 commits
    • Oran Agra's avatar
      Redis 6.2.13 · 24c0bf5c
      Oran Agra authored
      24c0bf5c
    • sundb's avatar
      Fix alpine tests (partial #12035) · b313fa41
      sundb authored
      This is a partial cherry-pick from Redis 7.2
      
      1) Fixed `ps -p [pid]`  doesn't output `<defunct>` when using procps 4.x causing `replication
        child dies when parent is killed - diskless` test to fail.
      
      (cherry picked from commit 42c8c618)
      (cherry picked from commit 63fc6c87071d12e3577fc383d5a3f25c60423c8e)
      b313fa41
    • sundb's avatar
      Re-enable hash downsize rehashing during fork (partial #12276) · f613873d
      sundb authored
      
      
      This was introduced by the recent change in #11692 which prevented a
      down-sizing rehashing while there is a fork.
      
      ## Solution
      1. Fix the rehashing code, so that the same as it allows rehashing for up-sizing
        during fork when the ratio is extreme, it will allow it for down-sizing as well.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      
      This is a partial cherry pick of:
      (cherry picked from commit b00a2351)
      (cherry picked from commit d4c37320382edb342292a3e30250d46896a12016)
      f613873d
    • Oran Agra's avatar
      Lua cjson and cmsgpack integer overflow issues (CVE-2022-24834) · 405d14fd
      Oran Agra authored
      
      
      * Fix integer overflows due to using wrong integer size.
      * Add assertions / panic when overflow still happens.
      * Deletion of dead code to avoid need to maintain it
      * Some changes are not because of bugs, but rather paranoia.
      * Improve cmsgpack and cjson test coverage.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      405d14fd
  3. 17 Apr, 2023 6 commits
    • Oran Agra's avatar
      Redis 6.22.12 · 2d4b9401
      Oran Agra authored
      2d4b9401
    • Oran Agra's avatar
      fix false valgrind error on new hash test (#11200) · 521e05cb
      Oran Agra authored
      New test fails on valgrind because strtold("+inf") with valgrind returns a non-inf result
      same thing is done in incr.tcl.
      
      (cherry picked from commit c3b7bde9)
      521e05cb
    • Binbin's avatar
      Fix the bug that CLIENT REPLY OFF|SKIP cannot receive push notifications (#11875) · 7afd1724
      Binbin authored
      This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
      mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
      With these flags, prepareClientToWrite called by addReply* will
      return C_ERR directly. So the client can't receive the Pub/Sub
      messages and any other push notifications, e.g client side tracking.
      
      In this PR, we adding a CLIENT_PUSHING flag, disables the reply
      silencing flags. When adding push replies, set the flag, after the reply,
      clear the flag. Then add the flag check in prepareClientToWrite.
      
      Fixes #11874
      
      Note, the SUBSCRIBE command response is a bit awkward,
      see https://github.com/redis/redis-doc/pull/2327
      
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 416842e6)
      (cherry picked from commit f8ae7a41)
      7afd1724
    • Madelyn Olson's avatar
      Always compact nodes in stream listpacks after creating new nodes (#11885) · eee173a6
      Madelyn Olson authored
      This change attempts to alleviate a minor memory usage degradation for Redis 6.2 and onwards when using rather large objects (~2k) in streams. Introduced in #6281, we pre-allocate the head nodes of a stream to be 4kb, to limit the amount of unnecessary initial reallocations that are done. However, if we only ever allocate one object because 2 objects exceeds the max_stream_entry_size, we never actually shrink it to fit the single item. This can lead to a lot of excessive memory usage. For smaller item sizes this becomes less of an issue, as the overhead decreases as the items become smaller in size.
      
      This commit also changes the MEMORY USAGE of streams, since it was reporting the lpBytes instead of the allocated size. This introduced an observability issue when diagnosing the memory issue, since Redis reported the same amount of used bytes pre and post change, even though the new implementation allocated more memory.
      
      (cherry picked from commit 2bb29e4a)
      (cherry picked from commit 17181517)
      eee173a6
    • Slava Koyfman's avatar
      Disconnect pub-sub subscribers when revoking `allchannels` permission (#11992) · 12634853
      Slava Koyfman authored
      The existing logic for killing pub-sub clients did not handle the `allchannels`
      permission correctly. For example, if you:
      
          ACL SETUSER foo allchannels
      
      Have a client authenticate as the user `foo` and subscribe to a channel, and then:
      
          ACL SETUSER foo resetchannels
      
      The subscribed client would not be disconnected, though new clients under that user
      would be blocked from subscribing to any channels.
      
      This was caused by an incomplete optimization in `ACLKillPubsubClientsIfNeeded`
      checking whether the new channel permissions were a strict superset of the old ones.
      
      (cherry picked from commit f38aa6bf)
      (cherry picked from commit 9caeadb8661f5fabd8386e3f07d445fe6ca46e7f)
      12634853
    • chendianqiang's avatar
      fix hincrbyfloat not to create a key if the new value is invalid (#11149) · e030e351
      chendianqiang authored
      
      
      Check the validity of the value before performing the create operation,
      prevents new data from being generated even if the request fails to execute.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarchendianqiang <chendianqiang@meituan.com>
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      (cherry picked from commit bc7fe41e)
      (cherry picked from commit 606a385935363ea46c0df4f40f8a949d85f7a20a)
      e030e351
  4. 28 Feb, 2023 8 commits
    • Oran Agra's avatar
      Redis 6.2.11 · 720ea82e
      Oran Agra authored
      720ea82e
    • Oran Agra's avatar
      Integer Overflow in RAND commands can lead to assertion (CVE-2023-25155) · 0a8a45f9
      Oran Agra authored
      Issue happens when passing a negative long value that greater than
      the max positive value that the long can store.
      
      (cherry picked from commit 41430af6a821c551abb862666ef896f2c196dea6)
      0a8a45f9
    • Tom Levy's avatar
      String pattern matching had exponential time complexity on pathological patterns (CVE-2022-36021) · f44b6a0e
      Tom Levy authored
      Authenticated users can use string matching commands with a
      specially crafted pattern to trigger a denial-of-service attack on Redis,
      causing it to hang and consume 100% CPU time.
      
      (cherry picked from commit e75f92047c22e659d49bba3a083cd0c9935f21e6)
      f44b6a0e
    • Madelyn Olson's avatar
      Backport client pause cron from #9549 · 1797c5e3
      Madelyn Olson authored
      1797c5e3
    • Madelyn Olson's avatar
      Prevent Redis from crashing from key tracking invalidations (#11814) · 683a4ce4
      Madelyn Olson authored
      (cherry picked from commit f7150c45bc5d6f03c8ba86a9a9296d024c6848dc)
      683a4ce4
    • zhaozhao.zz's avatar
      correct cluster inbound link keepalive time (#11785) · 2322f3d8
      zhaozhao.zz authored
      (cherry picked from commit a35e0837)
      (cherry picked from commit 76473f50990e06872d5a08886549815077f5def5)
      2322f3d8
    • uriyage's avatar
      Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766) · f084778c
      uriyage authored
      
      
      In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation
      size in order to:
      
      > reduce the need for realloc calls by making the sds implicitly take over
      the internal fragmentation
      
      This change was done most sds functions, excluding `sdsRemoveFreeSpace` and
      `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer)
      we call sdsRemoveFreeSpace when we see excessive free space and want to trim it.
      so if we don't trim it exactly to size, the caller may still see excessive free space and
      call it again and again.
      
      However, this resulted in some excessive calls to realloc, even when there's no need
      and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13).
      
      It turns out that a call for realloc with jemalloc can be expensive even if it ends up
      doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid
      the call for realloc.
      
      in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common
      code. the difference between them was that sdsResize would avoid using SDS_TYPE_5,
      since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace
      would permit using SDS_TYPE_5 and get an optimal memory consumption.
      now both methods take a `would_regrow` argument that makes it more explicit.
      
      the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize
      and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both.
      
      The new test that was added to cover this concern used to pass before this PR as well,
      this PR is just a performance optimization and cleanup.
      
      Benchmark:
      `redis-benchmark -c 100 -t set  -d 512 -P 10  -n  100000000`
      on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs).
      some 4.5% improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 46393f98)
      (cherry picked from commit b12eeccddd9318a5d97a5aee2dad88999dfad53f)
      f084778c
    • Oran Agra's avatar
      Make sure that fork child doesn't do incremental rehashing (#11692) · 658f9bf8
      Oran Agra authored
      Turns out that a fork child calling getExpire while persisting keys (and
      possibly also a result of some module fork tasks) could cause dictFind
      to do incremental rehashing in the child process, which is both a waste
      of time, and also causes COW harm.
      
      (cherry picked from commit 2bec254d)
      (cherry picked from commit 3e82bdf7)
      658f9bf8
  5. 17 Jan, 2023 2 commits
    • Oran Agra's avatar
      Redis 6.2.10 · 9c76b1ab
      Oran Agra authored
      9c76b1ab
    • Oran Agra's avatar
      Revert change to KEYS command from (#11676) · e12aacf3
      Oran Agra authored
      in Redis 7.0 this fix covers KEYS as well, but in 6.2 and 6.0 it doesn't,
      this is because in 7.0 there's a mechanism to avoid sending partial replies
      to the client, and in older releases there isn't, and without it there's a
      risk that the client would be able to read what looks like a complete KEYS
      command.
      e12aacf3
  6. 16 Jan, 2023 5 commits
    • Oran Agra's avatar
      Redis 6.2.9 · 137696d8
      Oran Agra authored
      137696d8
    • Oran Agra's avatar
      Fix range issues in ZRANDMEMBER and HRANDFIELD (CVE-2023-22458) · 37b3b2a7
      Oran Agra authored
      missing range check in ZRANDMEMBER and HRANDIFLD leading to panic due
      to protocol limitations
      37b3b2a7
    • Oran Agra's avatar
      Avoid integer overflows in SETRANGE and SORT (CVE-2022-35977) · 54538998
      Oran Agra authored
      Authenticated users issuing specially crafted SETRANGE and SORT(_RO)
      commands can trigger an integer overflow, resulting with Redis attempting
      to allocate impossible amounts of memory and abort with an OOM panic.
      54538998
    • Oran Agra's avatar
      Obuf limit, exit during loop in *RAND* commands and KEYS · 3148f3e8
      Oran Agra authored
      Related to the hang reported in #11671
      Currently, redis can disconnect a client due to reaching output buffer limit,
      it'll also avoid feeding that output buffer with more data, but it will keep
      running the loop in the command (despite the client already being marked for
      disconnection)
      
      This PR is an attempt to mitigate the problem, specifically for commands that
      are easy to abuse, specifically: KEYS, HRANDFIELD, SRANDMEMBER, ZRANDMEMBER.
      The RAND family of commands can take a negative COUNT argument (which is not
      bound to the number of elements in the key), so it's enough to create a key
      with one field, and then these commands can be used to hang redis.
      For KEYS the caller can use the existing keyspace in redis (if big enough).
      3148f3e8
    • Moti Cohen's avatar
      Fix sentinel issue if replica changes IP (#11590) · 3ebace93
      Moti Cohen authored
      
      
      As Sentinel supports dynamic IP only when using hostnames, there
      are few leftover addess comparison logic that doesn't take into
      account that the IP might get change.
      Co-authored-by: default avatarmoticless <moticless@github.com>
      (cherry picked from commit 4a27aa48)
      3ebace93
  7. 12 Dec, 2022 11 commits
    • Oran Agra's avatar
      Redis 6.2.8 · 423c78f4
      Oran Agra authored
      423c78f4
    • Yossi Gottlieb's avatar
      Fix TLS tests on newer tcl-tls/OpenSSL. (#10910) · de4d78d7
      Yossi Gottlieb authored
      Before this commit, TLS tests on Ubuntu 22.04 would fail as dropped
      connections result with an ECONNABORTED error thrown instead of an empty
      read.
      
      (cherry picked from commit 69d55768)
      de4d78d7
    • Yossi Gottlieb's avatar
      Use 'gcc' instead of 'ld' to link test modules. (#9710) · e5e642aa
      Yossi Gottlieb authored
      This solves several problems in a more elegant way:
      
      * No need to explicitly use `-lc` on x86_64 when building with `-m32`.
      * Avoids issues with undefined floating point emulation funcs on ARM.
      
      (cherry picked from commit f26e90be)
      e5e642aa
    • Yossi Gottlieb's avatar
      Fix daily failures due to macos-latest change. (#9637) · 52cccfbe
      Yossi Gottlieb authored
      * Fix test modules linking on macOS 11.x.
      * Use macOS 10.x for FreeBSD VM as VirtualBox is not yet supported on
        11.
      
      (cherry picked from commit 6d5a9117)
      52cccfbe
    • Ozan Tezcan's avatar
      Some fixes to undefined behaviour bugs taken from (#9601) · 052e01e7
      Ozan Tezcan authored
      **Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit.
      
      partial cherry pick from commit b91d8b28
      The bug in BITFIELD seems to affect 12.2.1 used on Alpine
      052e01e7
    • Madelyn Olson's avatar
      Initialize manual failover replica target (#9814) · 5d66aa3d
      Madelyn Olson authored
      (cherry picked from commit 362b3b02)
      5d66aa3d
    • Oran Agra's avatar
      Bump vmactions/freebsd-vm to 0.3.0 · 2f94d40e
      Oran Agra authored
      2f94d40e
    • Oran Agra's avatar
      resolve build warnings in quicklist test · d345d1da
      Oran Agra authored
      d345d1da
    • Oran Agra's avatar
      Try to fix a race in psync2 test (#11553) · 0a3ae0f0
      Oran Agra authored
      This test sets the master ping interval to 1 hour, in order to avoid
      pings in the replicatoin stream incrementing the replication offset,
      however, it didn't increase the repl-timeout so on slow machines
      where the test took more than 60 seconds, the replicas would drop
      and reconnect.
      
      ```
      *** [err]: PSYNC2: Partial resync after restart using RDB aux fields in tests/integration/psync2.tcl
      Replica didn't partial sync
      ```
      
      The test would detect 4 additional partial syncs where it expects
      only one.
      
      (cherry picked from commit b0250b45)
      0a3ae0f0
    • chenyang8094's avatar
      Fix null pointer subtraction warning (#10498) · 6029bd56
      chenyang8094 authored
      The warning:
      ```
      pqsort.c:106:7: warning: performing pointer subtraction with a null pointer has undefined behavior [-Wnull-pointer-subtraction]
      loop:   SWAPINIT(a, es);
              ^~~~~~~~~~~~~~~
      pqsort.c:65:47: note: expanded from macro 'SWAPINIT'
      #define SWAPINIT(a, es) swaptype = ((char *)a - (char *)NULL) % sizeof(long) || \
      ```
      Clang version:
      ```
      Apple clang version 13.1.6 (clang-1316.0.21.2)
      Target: x86_64-apple-darwin21.3.0
      Thread model: posix
      InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
      ```
      
      (cherry picked from commit cb625844)
      6029bd56
    • uriyage's avatar
      Module CLIENT_CHANGE, Fix crash on free blocked client with DB!=0 (#11500) · 39e5a6fa
      uriyage authored
      
      
      In moduleFireServerEvent we change the real client DB to 0 on freeClient in case the event is REDISMODULE_EVENT_CLIENT_CHANGE.
      It results in a crash if the client is blocked on a key on other than DB 0.
      
      The DB change is not necessary even for module-client, as we set its DB to 0 on either createClient or moduleReleaseTempClient.
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      Co-authored-by: default avatarBinbin <binloveplay1314@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit e4eb18b3)
      39e5a6fa