1. 21 Jul, 2021 12 commits
    • Binbin's avatar
      Fix accidental deletion of sinterstore command when we meet wrong type error. (#9032) · 1655576e
      Binbin authored
      SINTERSTORE would have deleted the dest key right away,
      even when later on it is bound to fail on an (WRONGTYPE) error.
      
      With this change it first picks up all the input keys, and only later
      delete the dest key if one is empty.
      
      Also add more tests for some commands.
      Mainly focus on
      - `wrong type error`:
      	expand test case (base on sinter bug) in non-store variant
      	add tests for store variant (although it exists in non-store variant, i think it would be better to have same tests)
      - the dstkey result when we meet `non-exist key (empty set)` in *store
      
      sdiff:
      - improve test case about wrong type error (the one we found in sinter, although it is safe in sdiff)
      - add test about using non-exist key (treat it like an empty set)
      sdiffstore:
      - according to sdiff test case, also add some tests about `wrong type error` and `non-exist key`
      - the different is that in sdiffstore, we will consider the `dstkey` result
      
      sunion/sunionstore add more tests (same as above)
      
      sinter/sinterstore also same as above ...
      
      (cherry picked from commit b8a5da80)
      (cherry picked from commit f4702b8b7a7da6cc661ddb6744cb322bc92e3267)
      1655576e
    • Jason Elbaum's avatar
      Change return value type for ZPOPMAX/MIN in RESP3 (#8981) · 8c0f06c2
      Jason Elbaum authored
      When using RESP3, ZPOPMAX/ZPOPMIN should return nested arrays for consistency
      with other commands (e.g. ZRANGE).
      
      We do that only when COUNT argument is present (similarly to how LPOP behaves).
      for reasoning see https://github.com/redis/redis/issues/8824#issuecomment-855427955
      
      This is a breaking change only when RESP3 is used, and COUNT argument is present!
      
      (cherry picked from commit 7f342020)
      (cherry picked from commit caaad2d686b2af0d13fbeda414e2b70e57635b5c)
      8c0f06c2
    • Rob Snyder's avatar
      Fix ziplist length updates on bigendian platforms (#2080) · 32d923f8
      Rob Snyder authored
      Adds call to intrev16ifbe to ensure ZIPLIST_LENGTH is compared correctly
      
      (cherry picked from commit eaa52719)
      (cherry picked from commit 4c181230855bb3f92e93b32a48db52e69bd7e509)
      32d923f8
    • perryitay's avatar
      Fail EXEC command in case a watched key is expired (#9194) · 8df81c03
      perryitay authored
      
      
      There are two issues fixed in this commit:
      1. we want to fail the EXEC command in case there is a watched key that's logically
         expired but not yet deleted by active expire or lazy expire.
      2. we saw that currently cache time is update in every `call()` (including nested calls),
         this time is being also being use for the isKeyExpired comparison, we want to update
         the cache time only in the first call (execCommand)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit ac8b1df8)
      8df81c03
    • Huang Zhw's avatar
      Do not install a file event to send data to rewrite child when parent stop... · 78c16235
      Huang Zhw authored
      Do not install a file event to send data to rewrite child when parent stop sending diff to child in aof rewrite. (#8767)
      
      In aof rewrite, when parent stop sending data to child, if there is
      new rewrite data, aofChildWriteDiffData write event will be installed.
      Then this event is issued and deletes the file event without do anyting.
      This will happen over and over again until aof rewrite finish.
      
      This bug used to waste a few system calls per excessive wake-up
      (epoll_ctl and epoll_wait) per cycle, each cycle triggered by receiving
      a write command from a client.
      
      (cherry picked from commit cb961d8c)
      78c16235
    • Omer Shadmi's avatar
      Avoid exiting to allow diskless loading to recover from RDB short read on module AUX data (#9199) · 518fd2dc
      Omer Shadmi authored
      Currently a replica is able to recover from a short read (when diskless loading
      is enabled) and avoid crashing/exiting, replying to the master and then the rdb
      could be sent again by the master for another load attempt by the replica.
      There were a few scenarios that were not behaving similarly, such as when
      there is no end-of-file marker, or when module aux data failed to load, which
      should be allowed to occur due to a short read.
      
      (cherry picked from commit f06d782f)
      518fd2dc
    • Oran Agra's avatar
      Fix race in client side tracking (#9116) · d80c8711
      Oran Agra authored
      The `Tracking gets notification of expired keys` test in tracking.tcl
      used to hung in valgrind CI quite a lot.
      
      It turns out the reason is that with valgrind and a busy machine, the
      server cron active expire cycle could easily run in the same event loop
      as the command that created `mykey`, so that when they key got expired,
      there were two change events to broadcast, one that set the key and one
      that expired it, but since we used raxTryInsert, the client that was
      associated with the "last" change was the one that created the key, so
      the NOLOOP filtered that event.
      
      This commit adds a test that reproduces the problem by using lazy expire
      in a multi-exec which makes sure the key expires in the same event loop
      as the one that added it.
      
      (cherry picked from commit 9b564b52)
      d80c8711
    • Maxim Galushka's avatar
      redis-cli: support for REDIS_REPLY_SET in CSV and RAW output. (#7338) · 981953a6
      Maxim Galushka authored
      Fixes #6792. Added support of REDIS_REPLY_SET in raw and csv output of `./redis-cli`
      
      Test:
      
      run commands to test:
        ./redis-cli -3 --csv COMMAND
        ./redis-cli -3 --raw COMMAND
      
      Now they are returning resuts, were failing with: "Unknown reply type: 10" before the change.
      
      (cherry picked from commit 96bb0785)
      981953a6
    • YaacovHazan's avatar
      unregister AE_READABLE from the read pipe in backgroundSaveDoneHandlerSocket (#8991) · da127ed3
      YaacovHazan authored
      In diskless replication, we create a read pipe for the RDB, between the child and the parent.
      When we close this pipe (fd), the read handler also needs to be removed from the event loop (if it still registered).
      Otherwise, next time we will use the same fd, the registration will be fail (panic), because
      we will use EPOLL_CTL_MOD (the fd still register in the event loop), on fd that already removed from epoll_ctl
      
      (cherry picked from commit 501d7755)
      da127ed3
    • guybe7's avatar
      ReplicationCron: Prevent invalid access to freed pointer (#8799) · fe417b06
      guybe7 authored
      Fixes #8797 
      
      (cherry picked from commit a60016e0)
      fe417b06
    • guybe7's avatar
      Add a timeout mechanism for replicas stuck in fullsync (#8762) · ea0a3764
      guybe7 authored
      Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork
      child so that the parent is the one sending data to the replicas.
      This mechanism has an issue in which a hung replica will cause the master to wait
      for it to read the data sent to it forever, thus preventing the fork child from terminating
      and preventing the creations of any other forks.
      
      This PR adds a timeout mechanism, much like the ACK-based timeout,
      we disconnect replicas that aren't reading the RDB file fast enough.
      
      (cherry picked from commit d63d0260)
      ea0a3764
    • Oran Agra's avatar
      if diskless repl child is killed, make sure to reap the pid (#7742) · 18429ce5
      Oran Agra authored
      Starting redis 6.0 and the changes we made to the diskless master to be
      suitable for TLS, I made the master avoid reaping (wait3) the pid of the
      child until we know all replicas are done reading their rdb.
      
      I did that in order to avoid a state where the rdb_child_pid is -1 but
      we don't yet want to start another fork (still busy serving that data to
      replicas).
      
      It turns out that the solution used so far was problematic in case the
      fork child was being killed (e.g. by the kernel OOM killer), in that
      case there's a chance that we currently disabled the read event on the
      rdb pipe, since we're waiting for a replica to become writable again.
      and in that scenario the master would have never realized the child
      exited, and the replica will remain hung too.
      Note that there's no mechanism to detect a hung replica while it's in
      rdb transfer state.
      
      The solution here is to add another pipe which is used by the parent to
      tell the child it is safe to exit. this mean that when the child exits,
      for whatever reason, it is safe to reap it.
      
      Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was
      part of #6271 (Accelerate diskless master connections) but was dropped
      when that PR was rebased after the TLS fork/pipe changes (5a477946).
      Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK
      has chance to detect that the child exited, it should be the one to call
      it so that we don't have to wait for cron (server.hz) to do that.
      
      (cherry picked from commit 573246f7)
      18429ce5
  2. 01 Jun, 2021 5 commits
  3. 30 May, 2021 1 commit
  4. 03 May, 2021 9 commits
    • Oran Agra's avatar
      Redis 6.0.13 · 68e93c22
      Oran Agra authored
      68e93c22
    • Oran Agra's avatar
      Resolve nonsense static analysis warnings · adf8f6f6
      Oran Agra authored
      (cherry picked from commit fd7d51c3)
      adf8f6f6
    • Oran Agra's avatar
      Fix integer overflow in STRALGO LCS (CVE-2021-29477) · 394614a5
      Oran Agra authored
      An integer overflow bug in Redis version 6.0 or newer could be exploited using
      the STRALGO LCS command to corrupt the heap and potentially result with remote
      code execution.
      
      (cherry picked from commit f0c5f920)
      394614a5
    • Oran Agra's avatar
      Fix integer overflow in intset (CVE-2021-29478) · 789f1015
      Oran Agra authored
      An integer overflow bug in Redis 6.2 could be exploited to corrupt the heap and
      potentially result with remote code execution.
      
      The vulnerability involves changing the default set-max-intset-entries
      configuration value, creating a large set key that consists of integer values
      and using the COPY command to duplicate it.
      
      The integer overflow bug exists in all versions of Redis starting with 2.6,
      where it could result with a corrupted RDB or DUMP payload, but not exploited
      through COPY (which did not exist before 6.2).
      
      (cherry picked from commit 29900d4e)
      789f1015
    • Igor's avatar
      Introduce fast path to bypass expensive serveClientsBlockedOnKeyByModule call (#8689) · 5c6f0b26
      Igor authored
      Introduce fast path to bypass expensive serveClientsBlockedOnKeyByModule call
      
      (cherry picked from commit cf0a909e)
      5c6f0b26
    • Madelyn Olson's avatar
      Skip unecessary check for pong recieved (#8585) · 8bcca06c
      Madelyn Olson authored
      
      (cherry picked from commit 98d2e001)
      8bcca06c
    • Oran Agra's avatar
      Fix crash report killed by message (#8683) · aae2d6e3
      Oran Agra authored
      We sometimes see the crash report saying we were killed by a random
      process even in cases where the crash was spontanius in redis.
      for instance, crashes found by the corrupt-dump test.
      
      It looks like this si_pid is sometimes left uninitialized, and a good
      way to tell if the crash originated in redis or trigged by outside is to
      look at si_code, real signal codes are always > 0, and ones generated by
      kill are have si_code of 0 or below.
      
      (cherry picked from commit b45b0d81)
      aae2d6e3
    • Yossi Gottlieb's avatar
      Server won't start on alpine/libmusl without IPv6. (#8655) · 94694010
      Yossi Gottlieb authored
      listenToPort attempts to gracefully handle and ignore certain errors but does not store errno prior to logging, which in turn calls several libc functions that may overwrite errno.
      
      This has been discovered due to libmusl strftime() always returning with errno set to EINVAL, which resulted with docker-library/redis#273.
      
      (cherry picked from commit df5f543b)
      94694010
    • guybe7's avatar
      Fix edge-case when a module client is unblocked (#8618) · 1b03a0d7
      guybe7 authored
      Scenario:
      1. A module client is blocked on keys with a timeout
      2. Shortly before the timeout expires, the key is being populated and signaled
         as ready
      3. Redis calls moduleTryServeClientBlockedOnKey (which replies to client) and
         then moduleUnblockClient
      4. moduleUnblockClient doesn't really unblock the client, it writes to
         server.module_blocked_pipe and only marks the BC as unblocked.
      5. beforeSleep kics in, by this time the client still exists and techincally
         timed-out. beforeSleep replies to the timeout client (double reply) and
         only then moduleHandleBlockedClients is called, reading from module_blocked_pipe
         and calling unblockClient
      
      The solution is similar to what was done in moduleTryServeClientBlockedOnKey: we
      should avoid re-processing an already-unblocked client
      
      (cherry picked from commit e58118cd)
      1b03a0d7
  5. 02 Mar, 2021 4 commits
  6. 22 Feb, 2021 9 commits
    • Oran Agra's avatar
      Redis 6.0.11 · 91ffe99e
      Oran Agra authored
      91ffe99e
    • Yossi Gottlieb's avatar
      Fix integer overflow (CVE-2021-21309). (#8522) · c9928576
      Yossi Gottlieb authored
      On 32-bit systems, setting the proto-max-bulk-len config parameter to a high value may result with integer overflow and a subsequent heap overflow when parsing an input bulk (CVE-2021-21309).
      
      This fix has two parts:
      
      Set a reasonable limit to the config parameter.
      Add additional checks to prevent the problem in other potential but unknown code paths.
      
      (cherry picked from commit d32f2e99)
      c9928576
    • Oran Agra's avatar
      Update CI on Ubuntu to tcl8.6 (since 20.04 is now used) (#8460) · 4bba84f0
      Oran Agra authored
      Github started shifting some repositoreis to use ubuntu 20.04 by default
      tcl8.5 is missing in these, but 8.6 exists in both 20.04 and 18.04
      
      (cherry picked from commit 018f7b73)
      4bba84f0
    • Viktor Söderqvist's avatar
      RM_ZsetRem: Delete key if empty (#8453) · 17c3ac89
      Viktor Söderqvist authored
      Without this fix, RM_ZsetRem can leave empty sorted sets which are
      not allowed to exist.
      
      Removing from a sorted set while iterating seems to work (while
      inserting causes failed assetions). RM_ZsetRangeEndReached is
      modified to return 1 if the key doesn't exist, to terminate
      iteration when the last element has been removed.
      
      (cherry picked from commit aea6e71e)
      17c3ac89
    • Greg Femec's avatar
      Fix random element selection for large hash tables. (#8133) · cde69883
      Greg Femec authored
      
      
      When a database on a 64 bit build grows past 2^31 keys, the underlying hash table expands to 2^32 buckets. After this point, the algorithms for selecting random elements only return elements from half of the available buckets because they use random() which has a range of 0 to 2^31 - 1. This causes problems for eviction policies which use dictGetSomeKeys or dictGetRandomKey. Over time they cause the hash table to become unbalanced because, while new keys are spread out evenly across all buckets, evictions come from only half of the available buckets. Eventually this half of the table starts to run out of keys and it takes longer and longer to find candidates for eviction. This continues until no more evictions can happen.
      
      This solution addresses this by using a 64 bit PRNG instead of libc random().
      Co-authored-by: default avatarGreg Femec <gfemec@google.com>
      (cherry picked from commit 266949c7)
      cde69883
    • Huang Zw's avatar
      Client tracking tracking-redir-broken push len is 2 not 3 (#8456) · e6a94aa5
      Huang Zw authored
      When redis responds with tracking-redir-broken push message (RESP3),
      it was responding with a broken protocol: an array of 3 elements, but only
      pushes 2 elements.
      
      (cherry picked from commit f687ac0c)
      e6a94aa5
    • filipe oliveira's avatar
      redis-benchmark: Fix broken protocol when used with -a or --dbnum (#8486) · 840b393c
      filipe oliveira authored
      Fix the pointers to the slot hash tags in the case of prefixed commands usage
      i.e. AUTH / SELECT
      
      It adjusts the pointers to the slot hash tags in the case of prefixed commands
      usage as soon as we get the 1st reply (same like we already did for the random
      strings within the command )
      
      (cherry picked from commit efccd635)
      840b393c
    • zhaozhao.zz's avatar
      XINFO should use lookupKeyReadOrReply (#8436) · 56ef0aa7
      zhaozhao.zz authored
      This bug would have let users observe logically expired keys on replicas and
      during CLIENT PAUSE WRITE.
      
      (cherry picked from commit f2a5fe36)
      56ef0aa7
    • Yossi Gottlieb's avatar
      Avoid assertions when testing arm64 cow bug. (#8405) · dcf409f8
      Yossi Gottlieb authored
      At least in one case the arm64 cow kernel bug test triggers an assert, which is a problem because it cannot be ignored like cases where the bug is found.
      
      On older systems (Linux <4.5) madvise fails because MADV_FREE is not supported. We treat these failures as an indication the system is not affected.
      
      Fixes #8351, #8406
      
      (cherry picked from commit 3a504904)
      dcf409f8