1. 21 Jul, 2021 5 commits
    • Huang Zhw's avatar
      On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS... · 5f49f4fb
      Huang Zhw authored
      On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191)
      
      GETBIT, SETBIT may access wrong address because of wrap.
      BITCOUNT and BITPOS may return wrapped results.
      BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761).
      
      This commit uses `uint64_t` or `long long` instead of `size_t`.
      related https://github.com/redis/redis/pull/8096
      
      At 32bit platform:
      > setbit bit 4294967295 1
      (integer) 0
      > config set proto-max-bulk-len 536870913
      OK
      > append bit "\xFF"
      (integer) 536870913
      > getbit bit 4294967296
      (integer) 0
      
      When the bit index is larger than 4294967295, size_t can't hold bit index. In the past,  `proto-max-bulk-len` is limit to 536870912, so there is no problem.
      
      After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct.
      
      For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow.
      But at 64bit platform, we don't have so long string. So this bug may never happen.
      
      Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.
      * This test is disabled in this version since bitops doesn't rely on
      proto-max-bulk-len. some of the overflows can still occur so we do want
      the fixes.
      
      (cherry picked from commit 71d45287)
      5f49f4fb
    • Oran Agra's avatar
      Test infra, handle RESP3 attributes and big-numbers and bools (#9235) · 62bc09d9
      Oran Agra authored
      - promote the code in DEBUG PROTOCOL to addReplyBigNum
      - DEBUG PROTOCOL ATTRIB skips the attribute when client is RESP2
      - networking.c addReply for push and attributes generate assertion when
        called on a RESP2 client, anything else would produce a broken
        protocol that clients can't handle.
      
      (cherry picked from commit 6a5bac30)
      (cherry picked from commit 7f38aa8bc719f709acdcefc35a45a7aa6faa76fa)
      62bc09d9
    • perryitay's avatar
      Fail EXEC command in case a watched key is expired (#9194) · 8df81c03
      perryitay authored
      
      
      There are two issues fixed in this commit:
      1. we want to fail the EXEC command in case there is a watched key that's logically
         expired but not yet deleted by active expire or lazy expire.
      2. we saw that currently cache time is update in every `call()` (including nested calls),
         this time is being also being use for the isKeyExpired comparison, we want to update
         the cache time only in the first call (execCommand)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit ac8b1df8)
      8df81c03
    • guybe7's avatar
      Add a timeout mechanism for replicas stuck in fullsync (#8762) · ea0a3764
      guybe7 authored
      Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork
      child so that the parent is the one sending data to the replicas.
      This mechanism has an issue in which a hung replica will cause the master to wait
      for it to read the data sent to it forever, thus preventing the fork child from terminating
      and preventing the creations of any other forks.
      
      This PR adds a timeout mechanism, much like the ACK-based timeout,
      we disconnect replicas that aren't reading the RDB file fast enough.
      
      (cherry picked from commit d63d0260)
      ea0a3764
    • Oran Agra's avatar
      if diskless repl child is killed, make sure to reap the pid (#7742) · 18429ce5
      Oran Agra authored
      Starting redis 6.0 and the changes we made to the diskless master to be
      suitable for TLS, I made the master avoid reaping (wait3) the pid of the
      child until we know all replicas are done reading their rdb.
      
      I did that in order to avoid a state where the rdb_child_pid is -1 but
      we don't yet want to start another fork (still busy serving that data to
      replicas).
      
      It turns out that the solution used so far was problematic in case the
      fork child was being killed (e.g. by the kernel OOM killer), in that
      case there's a chance that we currently disabled the read event on the
      rdb pipe, since we're waiting for a replica to become writable again.
      and in that scenario the master would have never realized the child
      exited, and the replica will remain hung too.
      Note that there's no mechanism to detect a hung replica while it's in
      rdb transfer state.
      
      The solution here is to add another pipe which is used by the parent to
      tell the child it is safe to exit. this mean that when the child exits,
      for whatever reason, it is safe to reap it.
      
      Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was
      part of #6271 (Accelerate diskless master connections) but was dropped
      when that PR was rebased after the TLS fork/pipe changes (5a477946).
      Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK
      has chance to detect that the child exited, it should be the one to call
      it so that we don't have to wait for cron (server.hz) to do that.
      
      (cherry picked from commit 573246f7)
      18429ce5
  2. 22 Feb, 2021 2 commits
    • Yossi Gottlieb's avatar
      CONFIG REWRITE should honor umask settings. (#8371) · 1237cce6
      Yossi Gottlieb authored
      Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one.
      
      The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly.
      
      Fixes #8369
      
      (cherry picked from commit b548ffab)
      1237cce6
    • Madelyn Olson's avatar
      Remove race condition and consistency issues with client pause and threaded IO (#8520) · 0c653428
      Madelyn Olson authored
      clientsArePaused isn't thread safe because it has a side effect of attempting to unpause,
      which may cause multiple threads concurrently updating the unblocked_clients global list.
      This change resolves this issue by no longer postponing client for threaded reads when
      clients are paused and then skipping the check for client paused for threaded reads,
      in case one is postponed and then clients are paused. (I don't think this is strictly possible,
      but being defensive seems better here)
      0c653428
  3. 12 Jan, 2021 5 commits
    • Yang Bodong's avatar
      Swapdb should make transaction fail if there is any client watching keys (#8239) · f464cf23
      Yang Bodong authored
      
      
      This PR not only fixes the problem that swapdb does not make the
      transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to
      set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients.
      
      FLUSHDB was changed to first iterate on all watched keys, and then on the
      clients watching each key.
      Instead of iterating though all clients, and for each iterate on watched keys.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 10f94b0a)
      f464cf23
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · 8ff02263
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b02780c4)
      
      in 6.0 this warning is ignored by default in order to avoid adding
      regression, specifically for deployments that don't...
      8ff02263
    • Wang Yuan's avatar
      Backup keys to slots map and restore when fail to sync if diskless-load type... · a60ed4a5
      Wang Yuan authored
      
      Backup keys to slots map and restore when fail to sync if diskless-load type is swapdb in cluster mode (#8108)
      
      When replica diskless-load type is swapdb in cluster mode, we didn't backup
      keys to slots map, so we will lose keys to slots map if fail to sync.
      Now we backup keys to slots map at first, and restore it properly when fail.
      
      This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit b55a827e)
      a60ed4a5
    • Oran Agra's avatar
      Fix oom-score-adj-values range, abs options, and bug when used in config file (#8046) · 3ac666b3
      Oran Agra authored
      Fix: When oom-score-adj-values is provided in the config file after
      oom-score-adj yes, it'll take an immediate action, before
      readOOMScoreAdj was acquired, resulting in an error (out of range score
      due to uninitialized value. delay the reaction the real call is made by
      main().
      
      Since the values are clamped to -1000..1000, and they're
      applied as an offset from the value at startup (which may be -1000), we
      need to allow the offsets to reach to +2000 so that a value of +1000 is
      achievable in case the value at startup was -1000.
      
      Adding an option for absolute values rather than relative ones.
      
      (cherry picked from commit 61954951)
      3ac666b3
    • guybe7's avatar
      EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016) · 4b37eb13
      guybe7 authored
      The bug was introduced by #5021 which only attempted avoid EXIST on an
      already expired key from returning 1 on a replica.
      
      Before that commit, dbExists was used instead of
      lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)
      
      Other than that, this commit fixes OBJECT to also come empty handed on
      expired keys in replica.
      
      And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
      the key regardless of it's expired state)
      
      (cherry picked from commit f8ae9917)
      4b37eb13
  4. 27 Oct, 2020 12 commits
    • Yossi Gottlieb's avatar
      Introduce getKeysResult for getKeysFromCommand. · 945983ac
      Yossi Gottlieb authored
      Avoid using a static buffer for short key index responses, and make it
      caller's responsibility to stack-allocate a result type. Responses that
      don't fit are still allocated on the heap.
      
      (cherry picked from commit 9b7f8ba8)
      945983ac
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · 76f3a63d
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      
      (cherry picked from commit bea40e6a)
      76f3a63d
    • Wang Yuan's avatar
      Set 'loading' and 'shutdown_asap' to volatile sig_atomic_t type (#7845) · 5d1ac2df
      Wang Yuan authored
      We may access and modify these two variables in signal handler function,
      to guarantee them async-signal-safe, so we should set them to volatile
      sig_atomic_t type.
      
      It doesn't look like this could have caused any real issue, and it seems that
      signals are handled in main thread on most platforms. But we want to follow C
      and POSIX standard in signal handler function.
      
      (cherry picked from commit f1863a1f)
      5d1ac2df
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 757ad7cd
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 57709c4b)
      757ad7cd
    • WuYunlong's avatar
      Make main thread killable so that it can be canceled at any time. · 4832cf4f
      WuYunlong authored
      Refine comment of makeThreadKillable().
      
      This commit can be backported to 5.0, only if we also backport 8b70cb0e
      
      .
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 647cac5b)
      4832cf4f
    • Oran Agra's avatar
      RM_GetContextFlags provides indication that we're in a fork child (#7783) · fd610ae0
      Oran Agra authored
      
      (cherry picked from commit 2458e548)
      fd610ae0
    • Daniel Dai's avatar
      fix make warnings in debug.c MacOS (#7805) · aa1feec7
      Daniel Dai authored
      
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 1b3b7520)
      aa1feec7
    • Wang Yuan's avatar
      Remove tmp rdb file in background thread (#7762) · 0bdddd3c
      Wang Yuan authored
      We're already using bg_unlink in several places to delete the rdb file in the background,
      and avoid paying the cost of the deletion from our main thread.
      This commit uses bg_unlink to remove the temporary rdb file in the background too.
      
      However, in case we delete that rdb file just before exiting, we don't actually wait for the
      background thread or the main thread to delete it, and just let the OS clean up after us.
      i.e. we open the file, unlink it and exit with the fd still open.
      
      Furthermore, rdbRemoveTempFile can be called from a thread and was using snprintf which is
      not async-signal-safe, we now use ll2string instead.
      
      (cherry picked from commit b002d2b4)
      0bdddd3c
    • Oran Agra's avatar
      Add printf attribute and fix warnings and a minor bug (#7803) · 7bc07260
      Oran Agra authored
      The fix in error handling of rdbGenericLoadStringObject is an actual bugfix
      
      (cherry picked from commit 092cfca5)
      7bc07260
    • WuYunlong's avatar
      bio: doFastMemoryTest should try to kill io threads as well. · ea4bf91a
      WuYunlong authored
      (cherry picked from commit f8660233)
      ea4bf91a
    • WuYunlong's avatar
      bio: fix doFastMemoryTest. · 5d933226
      WuYunlong authored
      If one thread got SIGSEGV, function sigsegvHandler() would be triggered,
      it would call bioKillThreads(). But call pthread_cancel() to cancel itself
      would make it block. Also note that if SIGSEGV is caught by bio thread, it
      should kill the main thread in order to give a positive report.
      
      (cherry picked from commit 8b70cb0e)
      5d933226
    • Oran Agra's avatar
      Squash merging 125 typo/grammar/comment/doc PRs (#7773) · 2ae7f491
      Oran Agra authored
      List of squashed commits or PRs
      ===============================
      
      commit 66801ea
      Author: hwware <wen.hui.ware@gmail.com>
      Date:   Mon Jan 13 00:54:31 2020 -0500
      
          typo fix in acl.c
      
      commit 46f55db
      Author: Itamar Haber <itamar@redislabs.com>
      Date:   Sun Sep 6 18:24:11 2020 +0300
      
          Updates a couple of comments
      
          Specifically:
      
          * RM_AutoMemory completed instead of pointing to docs
          * Updated link to custom type doc
      
      commit 61a2aa0
      Author: xindoo <xindoo@qq.com>
      Date:   Tue Sep 1 19:24:59 2020 +0800
      
          Correct errors in code comments
      
      commit a5871d1
      Author: yz1509 <pro-756@qq.com>
      Date:   Tue Sep 1 18:36:06 2020 +0800
      
          fix typos in module.c
      
      commit 41eede7
      Author: bookug <bookug@qq.com>
      Date:   Sat Aug 15 01:11:33 2020 +0800
      
          docs: fix typos in comments
      
      commit c303c84
      Author: lazy-snail <ws.niu@outlook.com>
      Date:   Fri Aug 7 11:15:44 2020 +0800
      
          fix spelling in redis.conf
      
      commit 1eb76bf
      Author: zhujian <zhujianxyz@gmail.com>...
      2ae7f491
  5. 10 Sep, 2020 2 commits
  6. 01 Sep, 2020 7 commits
  7. 20 Jul, 2020 3 commits
    • Yossi Gottlieb's avatar
      TLS: Session caching configuration support. (#7420) · 7a536c29
      Yossi Gottlieb authored
      * TLS: Session caching configuration support.
      * TLS: Remove redundant config initialization.
      
      (cherry picked from commit 3e6f2b1a)
      7a536c29
    • Oran Agra's avatar
      RESTORE ABSTTL won't store expired keys into the db (#7472) · 95ba01b5
      Oran Agra authored
      Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
      key and return success, RESTORE should not store key that are already
      expired into the db.
      When used together with REPLACE it should emit a DEL to keyspace
      notification and replication stream.
      
      (cherry picked from commit 5977a948)
      95ba01b5
    • Oran Agra's avatar
      EXEC always fails with EXECABORT and multi-state is cleared · 05e483cb
      Oran Agra authored
      In order to support the use of multi-exec in pipeline, it is important that
      MULTI and EXEC are never rejected and it is easy for the client to know if the
      connection is still in multi state.
      
      It was easy to make sure MULTI and DISCARD never fail (done by previous
      commits) since these only change the client state and don't do any actual
      change in the server, but EXEC is a different story.
      
      Since in the past, it was possible for clients to handle some EXEC errors and
      retry the EXEC, we now can't affort to return any error on EXEC other than
      EXECABORT, which now carries with it the real reason for the abort too.
      
      Other fixes in this commit:
      - Some checks that where performed at the time of queuing need to be re-
        validated when EXEC runs, for instance if the transaction contains writes
        commands, it needs to be aborted. there was one check that was already done
        in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
        -NOREPLICAS, -MASTERDOWN
      - When a command is rejected by processCommand it was rejected with addReply,
        which was not recognized as an error in case the bad command came from the
        master. this will enable to count or MONITOR these errors in the future.
      - make it easier for tests to create additional (non deferred) clients.
      - add tests for the fixes of this commit.
      
      (cherry picked from commit 65a3307b)
      05e483cb
  8. 12 Jun, 2020 2 commits
  9. 28 May, 2020 2 commits