1. 25 Nov, 2021 1 commit
    • uriyage's avatar
      Do not watch keys for dirty client (#9829) · fa48fb2d
      uriyage authored
      
      
      Currently, the watching clients are marked as dirty when a watched
      key is touched, but we continue watching the keys for no reason.
      Then, when the same key is touched again, we iterate again on the
      watching clients list and mark all clients as dirty again.
      Only when the exec/unwatch command is issued will the client be
      removed from the key->watching_clients list. The same applies when
      a dirty client calls the WATCH command. The key will be added to be
      watched by the client even if it has no effect.
      
      In the field, no performance degradation was observed as a result of the
      current implementation; it is merely a cleanup with possible memory and
      performance gains in some situations.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      fa48fb2d
  2. 24 Nov, 2021 2 commits
    • Pavel Melkozerov's avatar
      fix fob bad log messages in rdbSave (#9842) (#9843) · 9630ded3
      Pavel Melkozerov authored
      
      
      logs message prints wrong file is failed to open temporary file
      logs have error occurred in getcwd (uses same errno to report error)
      Co-authored-by: default avatarPavel Melkozerov <pavel.melkozerov@nokia.com>
      9630ded3
    • sundb's avatar
      Replace ziplist with listpack in quicklist (#9740) · 45129059
      sundb authored
      
      
      Part three of implementing #8702, following #8887 and #9366 .
      
      ## Description of the feature
      1. Replace the ziplist container of quicklist with listpack.
      2. Convert existing quicklist ziplists on RDB loading time. an O(n) operation.
      
      ## Interface changes
      1. New `list-max-listpack-size` config is an alias for `list-max-ziplist-size`.
      2. Replace `debug ziplist` command with `debug listpack`.
      
      ## Internal changes
      1. Add `lpMerge` to merge two listpacks . (same as `ziplistMerge`)
      2. Add `lpRepr` to print info of listpack which is used in debugCommand and `quicklistRepr`. (same as `ziplistRepr`)
      3. Replace `QUICKLIST_NODE_CONTAINER_ZIPLIST` with `QUICKLIST_NODE_CONTAINER_PACKED`(following #9357 ).
          It represent that a quicklistNode is a packed node, as opposed to a plain node.
      4. Remove `createZiplistObject` method, which is never used.
      5. Calculate listpack entry size using overhead overestimation in `quicklistAllowInsert`.
          We prefer an overestimation, which would at worse lead to a few bytes below the lowest limit of 4k.
      
      ## Improvements
      1. Calling `lpShrinkToFit` after converting Ziplist to listpack, which was missed at #9366.
      2. Optimize `quicklistAppendPlainNode` to avoid memcpy data.
      
      ## Bugfix
      1. Fix crash in `quicklistRepr` when ziplist is compressed, introduced from #9366.
      
      ## Test
      1. Add unittest for `lpMerge`.
      2. Modify the old quicklist ziplist corrupt dump test.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      45129059
  3. 23 Nov, 2021 2 commits
    • Oran Agra's avatar
      fix invalid read on corrupt ziplist (#9831) · a3a01429
      Oran Agra authored
      If the last bytes in ziplist are corrupt and we decode from tail to head,
      we may reach slightly outside the ziplist.
      a3a01429
    • guybe7's avatar
      QUIT is a command, HOST: and POST are not (#9798) · b161cff5
      guybe7 authored
      Some people complain that QUIT is missing from help/command table.
      Not appearing in COMMAND command, command stats, ACL, etc.
      and instead, there's a hack in processCommand with a comment that looks outdated.
      Note that it is [documented](https://redis.io/commands/quit)
      
      At the same time, HOST: and POST are there in the command table although these are not real commands.
      They would appear in the COMMAND command, and even in commandstats.
      
      Other changes:
      1. Initialize the static logged_time static var in securityWarningCommand
      2. add `no-auth` flag to RESET so it can always be executed.
      b161cff5
  4. 22 Nov, 2021 1 commit
    • Oran Agra's avatar
      Fix invalid access in lpFind on corrupted listpack (#9819) · f07dedf7
      Oran Agra authored
      Issue found by corrupt-dump-fuzzer test with ASAN.
      The problem was that lpSkip and lpGetWithSize could read the next listpack entry without validating that it's in range.
      Similarly even the memcmp in lpFind could do that and possibly crash on segfault and now they'll crash on assert first.
      
      The naive fix of using lpAssertValidEntry every time, resulted in 30% degradation in the lpFind benchmark of the unit test.
      The final fix with the condition at the bottom has no performance implications.
      f07dedf7
  5. 21 Nov, 2021 3 commits
    • Oran Agra's avatar
      Fix false positive leak reported by GCC ASAN (#9816) · 183b90a6
      Oran Agra authored
      Leak found by the corrupt-dump-fuzzer when using GCC ASAN, which seems
      to falsely report leaks on pointers kept only on the stack when calling exit.
      Instead we now use _exit on panic / assert to skip these leak checks.
      
      Additionally, check for sanitizer warnings in the corrupt-dump-fuzzer between iterations,
      so that when something is found we know which test to relate it too (and it prints reproduction command list)
      183b90a6
    • Yossi Gottlieb's avatar
      Fix occasional RM_Call() crashes. (#9805) · fd0ca747
      Yossi Gottlieb authored
      With dynamically growing argc (#9528), it is necessary to initialize
      argv_len. Normally createClient() handles that, but in the case of a
      module shared_client, this needs to be done explicitly.
      
      This also addresses an issue with rewriteClientCommandArgument() which
      doesn't properly handle the case where the new element extends beyond
      argc but not beyond argv_len.
      fd0ca747
    • Oran Agra's avatar
      Prevent LCS from allocating temp memory over proto-max-bulk-len (#9817) · 14176484
      Oran Agra authored
      LCS can allocate immense amount of memory (sizes of two inputs multiplied by each other).
      In the past this caused some possible security issues due to overflows, which we solved
      and also added use of `trymalloc` to return "Insufficient memory" instead of OOM panic zmalloc.
      
      But in case overcommit is enabled, it could be that we won't get the OOM panic, and zmalloc
      will succeed, and then we can get OOM killed by the kernel.
      
      The solution here is to prevent LCS from allocating transient memory that's bigger than
      `proto-max-bulk-len` config.
      This config is not directly related to transient memory, but using a hard coded value ad well as
      introducing a specific config seems wrong.
      
      This comes to solve an error in the corrupt-dump-fuzzer test that started in the daily CI see #9799
      14176484
  6. 20 Nov, 2021 1 commit
  7. 19 Nov, 2021 1 commit
  8. 18 Nov, 2021 6 commits
    • Wen Hui's avatar
      75d50e5d
    • perryitay's avatar
      Fix crashes when list-compress-depth is used. (#9779) · 0c10f0e1
      perryitay authored
      Recently we started using list-compress-depth in tests (was completely untested till now).
      Turns this triggered test failures with the external mode, since the tests left the setting enabled
      and then it was used in other tests (specifically the fuzzer named "Stress tester for #3343-alike bugs").
      
      This PR fixes the issue of the `recompress` flag being left set by mistake, which caused the code to
      later to compress the head or tail nodes (which should never be compressed)
      
      The solution is to reset the recompress flag when it should have been (when it was decided not to compress).
      
      Additionally we're adding some assertions and improve the tests so in order to catch other similar bugs.
      0c10f0e1
    • Eduardo Semprebon's avatar
      Reject PING with MASTERDOWN when replica-serve-stale-data=no (#9757) · 1a255e31
      Eduardo Semprebon authored
      Currently PING returns different status when server is not serving data,
      for example when `LOADING` or `BUSY`.
      But same was not true for `MASTERDOWN`
      This commit makes PING reply with `MASTERDOWN` when
      replica-serve-stale-data=no and link is MASTER is down.
      1a255e31
    • guybe7's avatar
      Obliterate STRALGO! add LCS (which only works on keys) (#9799) · af748988
      guybe7 authored
      Drop the STRALGO command, now LCS is a command of its own and it only works on keys (not input strings).
      The motivation is that STRALGO's syntax was really messed-up...
      - assumes all (future) string algorithms will take similar arguments
      - mixes command that takes keys and one that doesn't in the same command.
      - make it nearly impossible to expose the right key spec in COMMAND INFO (issues cluster clients)
      - hard for cluster clients to determine the key names (firstkey, lastkey, etc)
      - hard for ACL / flags (is it a read command?)
      
      This is a breaking change.
      af748988
    • Binbin's avatar
      Fixes ZPOPMIN/ZPOPMAX wrong replies when count is 0 with non-zset (#9711) · 91e77a0c
      Binbin authored
      Moves ZPOP ... 0 fast exit path after type check to reply with
      WRONGTYPE. In the past it will return an empty array.
      
      Also now count is not allowed to be negative.
      
      see #9680
      
      before:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (empty array)
      127.0.0.1:6379> zpopmin zset -1
      (empty array)
      ```
      
      after:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (error) WRONGTYPE Operation against a key holding the wrong kind of value
      127.0.0.1:6379> zpopmin zset -1
      (error) ERR value is out of range, must be positive
      ```
      91e77a0c
    • Madelyn Olson's avatar
      Unpause clients earlier during manual cluster failover (#9676) · 32215e78
      Madelyn Olson authored
      Unpause clients after manual failover ends instead of the timed offset
      32215e78
  9. 16 Nov, 2021 5 commits
    • Axlgrep's avatar
      optimze the judgment logic of use_pattern in scanGenericCommand() (#9789) · 5f8ee381
      Axlgrep authored
      in `scan 0 match ""` case,  pat is empty sds(patlen is 0), I don't think should access the
      first character directly in this case(even though the first character is ' \0 '), for the
      code readability, I switch the two positions of judgment logic. 
      5f8ee381
    • harleyliao's avatar
      set aof rewrite status err, when fork fail (#5606) · 992a610f
      harleyliao authored
      when aof rewrite is failed by fork(), It'll be indicated by aof_last_bgrewrite_status
      INFO field, same as when the fork child fails later on.
      992a610f
    • sundb's avatar
      Change lzf to handle values larger than UINT32_MAX (#9776) · 985430b4
      sundb authored
      Redis supports inserting data over 4GB into string (and recently for lists too, see #9357),
      But LZF compression used in RDB files (see `rdbcompression` config), and in quicklist
      (see `list-compress-depth` config) does not support compress/decompress data over
      UINT32_MAX, which will result in corrupting the rdb after compression.
      
      Internal changes:
      1. Modify the `unsigned int` parameter of `lzf_compress/lzf_decompress` to `size_t`.
      2. Modify the variable types in `lzf_compress` involving offsets and lengths to `size_t`.
      3. Set LZF_USE_OFFSETS to 0.
          When LZF_USE_OFFSETS is 1, lzf store offset into `LZF_HSLOT`(32bit). 
          Even in 64-bit, `LZF_USE_OFFSETS` defaults to 1, because lzf assumes that it only
          compresses and decompresses data smaller than UINT32_MAX.
          But now we need to make lzf support 64-bit, turning on `LZF_USE_OFFSETS` will make
          it impossible to store 64-bit offsets or pointers.
          BTW, disable LZF_USE_OFFSETS also brings a few performance improvements.
      
      Tests:
      1. Add test for compress/decompress string large than UINT32_MAX.
      2. Add unittest for compress/decompress quicklistNode.
      985430b4
    • sundb's avatar
      Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) · e725d737
      sundb authored
      This is a preparation step in order to add a new test in quicklist.c see #9776
      e725d737
    • guoxiang1996's avatar
      insufficient size for cached client flags in call() (#9783) · aba70df4
      guoxiang1996 authored
      The client flags is a 64 bit integer, but the temporary cached value on the stack of call() is 32 bit.
      luckily this doesn't lead to any bugs since the only flags used against this variables are below 32 bit.
      aba70df4
  10. 11 Nov, 2021 1 commit
    • Ozan Tezcan's avatar
      Add sanitizer support and clean up sanitizer findings (#9601) · b91d8b28
      Ozan Tezcan authored
      - Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
      - To build Redis with desired sanitizer : `make SANITIZER=undefined`
      - There were some sanitizer findings, cleaned up codebase
      - Added tests with address and undefined behavior sanitizers to daily CI.
      - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).
      
      Basically, there are three types of issues : 
      
      **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
      does not support unaligned access. Redis does unaligned access only on supported platforms.
      
      **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
       **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit. 
      b91d8b28
  11. 09 Nov, 2021 2 commits
  12. 08 Nov, 2021 3 commits
  13. 07 Nov, 2021 1 commit
    • yoav-steinberg's avatar
      Refactor config.c for generic setter interface (#9644) · 79ac5756
      yoav-steinberg authored
      
      
      This refactors all `CONFIG SET`s and conf file loading arguments go through
      the generic config handling interface.
      
      Refactoring changes:
      - All config params go through the `standardConfig` interface (some stuff which
        is only related to the config file and not the `CONFIG` command still has special
        handling for rewrite/config file parsing, `loadmodule`, for example.) .
      - Added `MULTI_ARG_CONFIG` flag for configs to signify they receive a variable
        number of arguments instead of a single argument. This is used to break up space
        separated arguments to `CONFIG SET` so the generic setter interface can pass
        multiple arguments to the setter function. When parsing the config file we also break
        up anything after the config name into multiple arguments to the setter function.
      
      Interface changes:
      - A side effect of the above interface is that the `bind` argument in the config file can
        be empty (no argument at all) this is treated the same as passing an single empty
        string argument (same as `save` already used to work).
      - Support rewrite and setting `watchdog-period` from config file (was only supported
        by the CONFIG command till now).
      - Another side effect is that the `save T X` config argument now supports multiple
        Time-Changes pairs in a single line like its `CONFIG SET` counterpart. So in the
        config file you can either do:
        ```
        save 3600 1
        save 600 10
        ```
        or do
        ```
        save 3600 1 600 10
        ```
      Co-authored-by: default avatarBjorn Svensson <bjorn.a.svensson@est.tech>
      79ac5756
  14. 05 Nov, 2021 1 commit
  15. 04 Nov, 2021 3 commits
    • Eduardo Semprebon's avatar
      Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) · 91d0c758
      Eduardo Semprebon authored
      
      
      For diskless replication in swapdb mode, considering we already spend replica memory
      having a backup of current db to restore in case of failure, we can have the following benefits
      by instead swapping database only in case we succeeded in transferring db from master:
      
      - Avoid `LOADING` response during failed and successful synchronization for cases where the
        replica is already up and running with data.
      - Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
        time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
      - This could be implemented also for disk replication with similar benefits if consumers are willing
        to spend the extra memory usage.
      
      General notes:
      - The concept of `backupDb` becomes `tempDb` for clarity.
      - Async loading mode will only kick in if the replica is syncing from a master that has the same
        repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline. 
      - New property in INFO: `async_loading` to differentiate from the blocking loading
      - Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
        and the tempDb that is passed around.
      - Because this is affecting replicas only, we assume that if they are not readonly and write commands
        during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
        here anyways to avoid complications.
      
      Considerations for review:
      - We have many cases where server.loading flag is used and even though I tried my best, there may
        be cases where async_loading should be checked as well and cases where it shouldn't (would require
        very good understanding of whole code)
      - Several places that had different behavior depending on the loading flag where actually meant to just
        handle commands coming from the AOF client differently than ones coming from real clients, changed
        to check CLIENT_ID_AOF instead.
      
      **Additional for Release Notes**
      - Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
        contribute on triggering next database SAVE
      - New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
      - Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
        Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
        ABORTED and COMPLETED.
      - New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
        to allow modules to declare they support the diskless replication with async loading (when absent, we fall
        back to disk-based loading).
      Co-authored-by: default avatarEduardo Semprebon <edus@saxobank.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      91d0c758
    • Itamar Haber's avatar
      Fixes LPOP/RPOP wrong replies when count is 0 (#9692) · 06dd202a
      Itamar Haber authored
      Introduced in #8179, this fixes the command's replies in the 0 count edge case.
      [BREAKING] changes the reply type when count is 0 to an empty array (instead of nil)
      Moves LPOP ... 0 fast exit path after type check to reply with WRONGTYPE
      06dd202a
    • menwen's avatar
      Retry when a blocked connection system call is interrupted by a signal (#9629) · ccf8a651
      menwen authored
      
      
      When repl-diskless-load is enabled, the connection is set to the blocking state.
      The connection may be interrupted by a signal during a system call.
      This would have resulted in a disconnection and possibly a reconnection loop.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ccf8a651
  16. 03 Nov, 2021 3 commits
    • perryitay's avatar
      Add support for list type to store elements larger than 4GB (#9357) · f27083a4
      perryitay authored
      
      
      Redis lists are stored in quicklist, which is currently a linked list of ziplists.
      Ziplists are limited to storing elements no larger than 4GB, so when bigger
      items are added they're getting truncated.
      This PR changes quicklists so that they're capable of storing large items
      in quicklist nodes that are plain string buffers rather than ziplist.
      
      As part of the PR there were few other changes in redis: 
      1. new DEBUG sub-commands: 
         - QUICKLIST-PACKED-THRESHOLD - set the threshold of for the node type to
           be plan or ziplist. default (1GB)
         - QUICKLIST <key> - Shows low level info about the quicklist encoding of <key>
      2. rdb format change:
         - A new type was added - RDB_TYPE_LIST_QUICKLIST_2 . 
         - container type (packed / plain) was added to the beginning of the rdb object
           (before the actual node list).
      3. testing:
         - Tests that requires over 100MB will be by default skipped. a new flag was
           added to 'runtest' to run the large memory tests (not used by default)
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      f27083a4
    • guybe7's avatar
      Fix COMMAND GETKEYS on EVAL without keys (#9733) · f11a2d4d
      guybe7 authored
      Add new no-mandatory-keys flag to support COMMAND GETKEYS of commands
      which have no mandatory keys.
      
      In the past we would have got this error:
      ```
      127.0.0.1:6379> command getkeys eval "return 1" 0
      (error) ERR Invalid arguments specified for command
      ```
      f11a2d4d
    • perryitay's avatar
      fix: lookupKey on SETNX and SETXX only once (#9640) · 77d3c6bf
      perryitay authored
      When using SETNX and SETXX we could end up doing key lookup twice.
      This presents a small inefficiency price.
      Also once we have statistics of write hit and miss they'll be wrong (recording the same key hit twice) 
      77d3c6bf
  17. 02 Nov, 2021 3 commits
  18. 01 Nov, 2021 1 commit
    • Oran Agra's avatar
      fix valgrind issues with long double module test (#9709) · f1f3cceb
      Oran Agra authored
      The module test in reply.tcl was introduced by #8521 but didn't run until recently (see #9639)
      and then it started failing with valgrind.
      This is because valgrind uses 64 bit long double (unlike most other platforms that have at least 80 bits)
      But besides valgrind, the tests where also incompatible with ARM32, which also uses 64 bit long doubles.
      
      We now use appropriate value to avoid issues with either valgrind or ARM32
      
      In all the double tests, i use 3.141, which is safe since since addReplyDouble uses
      `%.17Lg` which is able to represent this value without adding any digits due to precision loss. 
      
      In the long double, since we use `%.17Lf` in ld2string, it preserves 17 significant
      digits, rather than 17 digit after the decimal point (like in `%.17Lg`).
      So to make these similar, i use value lower than 1 (no digits left of
      the period)
      
      Lastly, we have the same issue with TCL (no long doubles) so we read
      raw protocol in that test.
      
      Note that the only error before this fix (in both valgrind and ARM32 is this:
      ```
      *** [err]: RM_ReplyWithLongDouble: a float reply in tests/unit/moduleapi/reply.tcl
      Expected '3.141' to be equal to '3.14100000000000001' (context: type eval line 2 cmd {assert_equal 3.141 [r rw.longdouble 3.141]} proc ::test)
      ```
      so the changes to debug.c and scripting.tcl aren't really needed, but i consider them a cleanup
      (i.e. scripting.c validated a different constant than the one that's sent to it from debug.c).
      
      Another unrelated change is to add the RESP version to the repeated tests in reply.tcl
      f1f3cceb