1. 24 Nov, 2021 3 commits
    • sundb's avatar
      Replace ziplist with listpack in quicklist (#9740) · 45129059
      sundb authored
      
      
      Part three of implementing #8702, following #8887 and #9366 .
      
      ## Description of the feature
      1. Replace the ziplist container of quicklist with listpack.
      2. Convert existing quicklist ziplists on RDB loading time. an O(n) operation.
      
      ## Interface changes
      1. New `list-max-listpack-size` config is an alias for `list-max-ziplist-size`.
      2. Replace `debug ziplist` command with `debug listpack`.
      
      ## Internal changes
      1. Add `lpMerge` to merge two listpacks . (same as `ziplistMerge`)
      2. Add `lpRepr` to print info of listpack which is used in debugCommand and `quicklistRepr`. (same as `ziplistRepr`)
      3. Replace `QUICKLIST_NODE_CONTAINER_ZIPLIST` with `QUICKLIST_NODE_CONTAINER_PACKED`(following #9357 ).
          It represent that a quicklistNode is a packed node, as opposed to a plain node.
      4. Remove `createZiplistObject` method, which is never used.
      5. Calculate listpack entry size using overhead overestimation in `quicklistAllowInsert`.
          We prefer an overestimation, which would at worse lead to a few bytes below the lowest limit of 4k.
      
      ## Improvements
      1. Calling `lpShrinkToFit` after converting Ziplist to listpack, which was missed at #9366.
      2. Optimize `quicklistAppendPlainNode` to avoid memcpy data.
      
      ## Bugfix
      1. Fix crash in `quicklistRepr` when ziplist is compressed, introduced from #9366.
      
      ## Test
      1. Add unittest for `lpMerge`.
      2. Modify the old quicklist ziplist corrupt dump test.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      45129059
    • Binbin's avatar
      Wait for `asyn_loading` to stop in `short read` test (#9841) · fb4f7be2
      Binbin authored
      In #9323, when `repl-diskless-load` is enabled and set to `swapdb`,
      if the master replication ID hasn't changed, we can load data-set
      asynchronously, and serving read commands during the full resync.
      
      In `diskless loading short read` test, after a loading successfully,
      we will wait for the loading to stop and continue the for loop.
      
      After the introduction of `async_loading`, we also need to check it.
      Otherwise the next loop will start too soon, may trigger a timing issue.
      fb4f7be2
    • Binbin's avatar
      Add tests to cover EXPIRE overflow fix (#9839) · 9273d09d
      Binbin authored
      In #8287, some overflow checks have been added. But when
      `when *= 1000` overflows, it will become a positive number.
      And the check not able to catch it. The key will be added with
      a short expiration time and will deleted a few seconds later.
      
      In #9601, will check the overflow after `*=` and return an
      error first, and avoiding this situation.
      
      In this commit, added some tests to cover those code paths.
      Found it in #9825, and close it.
      9273d09d
  2. 23 Nov, 2021 2 commits
    • Oran Agra's avatar
      fix invalid read on corrupt ziplist (#9831) · a3a01429
      Oran Agra authored
      If the last bytes in ziplist are corrupt and we decode from tail to head,
      we may reach slightly outside the ziplist.
      a3a01429
    • guybe7's avatar
      QUIT is a command, HOST: and POST are not (#9798) · b161cff5
      guybe7 authored
      Some people complain that QUIT is missing from help/command table.
      Not appearing in COMMAND command, command stats, ACL, etc.
      and instead, there's a hack in processCommand with a comment that looks outdated.
      Note that it is [documented](https://redis.io/commands/quit)
      
      At the same time, HOST: and POST are there in the command table although these are not real commands.
      They would appear in the COMMAND command, and even in commandstats.
      
      Other changes:
      1. Initialize the static logged_time static var in securityWarningCommand
      2. add `no-auth` flag to RESET so it can always be executed.
      b161cff5
  3. 22 Nov, 2021 3 commits
    • Oran Agra's avatar
      Fix invalid access in lpFind on corrupted listpack (#9819) · f07dedf7
      Oran Agra authored
      Issue found by corrupt-dump-fuzzer test with ASAN.
      The problem was that lpSkip and lpGetWithSize could read the next listpack entry without validating that it's in range.
      Similarly even the memcmp in lpFind could do that and possibly crash on segfault and now they'll crash on assert first.
      
      The naive fix of using lpAssertValidEntry every time, resulted in 30% degradation in the lpFind benchmark of the unit test.
      The final fix with the condition at the bottom has no performance implications.
      f07dedf7
    • Oran Agra's avatar
      fix string escaping in corrupt-dump test to support TCL8.5 (#9824) · f00a8ad9
      Oran Agra authored
      TCL8.5 can't handle cases where part of the string is escaped and part of it isn't,
      if there's a single char that needs escaping, we need to escape the whole string.
      f00a8ad9
    • Binbin's avatar
      Fix timing issue in sub-second expires test (#9821) · 698b5774
      Binbin authored
      
      
      The `PEXPIRE/PSETEX/PEXPIREAT can set sub-second expires` test is
      a very time sensitive test, it used to occasionally fail on MacOS.
      
      It will perform there internal tests in a loop, as long as one
      fails, it will try to excute again in the next loop.
      
      oranagra suggested that we can split it into three individual tests,
      so that if one fails, we do not need to retry the others. And maybe
      it will increase the chances of success dramatically.
      
      Each is executed 500 times, and the number of retries is collected:
      ```
      PSETEX, total: 500, sum: 745, min: 0, max: 13, avg: 1.49
      
      PEXPIRE, total: 500, sum: 575, min: 0, max: 16, avg: 1.15
      
      PEXPIREAT, total: 500, sum: 0, min: 0, max: 0, avg: 0.0
      
      ALL(old_way), total: 500, sum: 8090, min: 0, max: 138, avg: 16.18
      ```
      
      And we can see the threshold is very low.
      Splitting the test also makes the code better to maintain.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      698b5774
  4. 21 Nov, 2021 5 commits
    • Oran Agra's avatar
      Fix false positive leak reported by GCC ASAN (#9816) · 183b90a6
      Oran Agra authored
      Leak found by the corrupt-dump-fuzzer when using GCC ASAN, which seems
      to falsely report leaks on pointers kept only on the stack when calling exit.
      Instead we now use _exit on panic / assert to skip these leak checks.
      
      Additionally, check for sanitizer warnings in the corrupt-dump-fuzzer between iterations,
      so that when something is found we know which test to relate it too (and it prints reproduction command list)
      183b90a6
    • Ozan Tezcan's avatar
      Don't use accurate option with ASAN unit tests (#9818) · a68b71ac
      Ozan Tezcan authored
      specifically the ziplist and listpack unit tests and benchmarks run for too long with address sanitizer and --accurate
      a68b71ac
    • Yossi Gottlieb's avatar
      Fix occasional RM_Call() crashes. (#9805) · fd0ca747
      Yossi Gottlieb authored
      With dynamically growing argc (#9528), it is necessary to initialize
      argv_len. Normally createClient() handles that, but in the case of a
      module shared_client, this needs to be done explicitly.
      
      This also addresses an issue with rewriteClientCommandArgument() which
      doesn't properly handle the case where the new element extends beyond
      argc but not beyond argv_len.
      fd0ca747
    • Oran Agra's avatar
      Prevent LCS from allocating temp memory over proto-max-bulk-len (#9817) · 14176484
      Oran Agra authored
      LCS can allocate immense amount of memory (sizes of two inputs multiplied by each other).
      In the past this caused some possible security issues due to overflows, which we solved
      and also added use of `trymalloc` to return "Insufficient memory" instead of OOM panic zmalloc.
      
      But in case overcommit is enabled, it could be that we won't get the OOM panic, and zmalloc
      will succeed, and then we can get OOM killed by the kernel.
      
      The solution here is to prevent LCS from allocating transient memory that's bigger than
      `proto-max-bulk-len` config.
      This config is not directly related to transient memory, but using a hard coded value ad well as
      introducing a specific config seems wrong.
      
      This comes to solve an error in the corrupt-dump-fuzzer test that started in the daily CI see #9799
      14176484
    • Oran Agra's avatar
      Improve active defrag in jemalloc 5.2 (#9778) · d4e7ffb3
      Oran Agra authored
      Background:
      Following the upgrade to jemalloc 5.2, there was a test that used to be flaky and
      started failing consistently (on 32bit), so we disabled it ​(see #9645).
      
      This is a test that i introduced in #7289 when i attempted to solve a rare stagnation
      problem, and it later turned out i failed to solve it, ans what's more i added a test that
      caused it to be not so rare, and as i mentioned, now in jemalloc 5.2 it became consistent on 32bit.
      
      Stagnation can happen when all the slabs of the bin are equally utilized, so the decision
      to move an allocation from a relatively empty slab to a relatively full one, will never
      happen, and in that test all the slabs are at 50% utilization, so the defragger could just
      keep scanning the keyspace and not move anything.
      
      What this PR changes:
      * First, finally in jemalloc 5.2 we have the count of non-full slabs, so when we compare
        the utilization of the current slab, we can compare it to the average utilization of the non-full
        slabs in our bin, instead of the total average of our bin. this takes the full slabs out of the game,
        since they're not candidates for migration (neither source nor target).
      * Secondly, We add some 12% (100/8) to the decision to defrag an allocation, this is the part
        that aims to avoid stagnation, and it's especially important since the above mentioned change
        can get us closer to stagnation.
      * Thirdly, since jemalloc 5.2 adds sharded bins, we take into account all shards (something
        that's missing from the original PR that merged it), this isn't expected to make any difference
        since anyway there should be just one shard.
      
      How this was benchmarked.
      What i did was run the memefficiency test unit with `--verbose` and compare the defragger hits
      and misses the tests reported.
      At first, when i took into consideration only the non-full slabs, it got a lot worse (i got into
      stagnation, or just got a lot of misses and a lot of hits), but when i added the 10% i got back
      to results that were slightly better than the ones of the jemalloc 5.1 branch. i.e. full defragmentation
      was achieved with fewer hits (relocations), and fewer misses (keyspace scans).
      d4e7ffb3
  5. 20 Nov, 2021 1 commit
  6. 19 Nov, 2021 1 commit
  7. 18 Nov, 2021 8 commits
    • Yossi Gottlieb's avatar
      366d5101
    • Wen Hui's avatar
      75d50e5d
    • perryitay's avatar
      Fix crashes when list-compress-depth is used. (#9779) · 0c10f0e1
      perryitay authored
      Recently we started using list-compress-depth in tests (was completely untested till now).
      Turns this triggered test failures with the external mode, since the tests left the setting enabled
      and then it was used in other tests (specifically the fuzzer named "Stress tester for #3343-alike bugs").
      
      This PR fixes the issue of the `recompress` flag being left set by mistake, which caused the code to
      later to compress the head or tail nodes (which should never be compressed)
      
      The solution is to reset the recompress flag when it should have been (when it was decided not to compress).
      
      Additionally we're adding some assertions and improve the tests so in order to catch other similar bugs.
      0c10f0e1
    • Binbin's avatar
      Add --dump-logs to ci tests and daily tests (#9803) · 020092e0
      Binbin authored
      So that we can print server log on test failure.
      020092e0
    • Eduardo Semprebon's avatar
      Reject PING with MASTERDOWN when replica-serve-stale-data=no (#9757) · 1a255e31
      Eduardo Semprebon authored
      Currently PING returns different status when server is not serving data,
      for example when `LOADING` or `BUSY`.
      But same was not true for `MASTERDOWN`
      This commit makes PING reply with `MASTERDOWN` when
      replica-serve-stale-data=no and link is MASTER is down.
      1a255e31
    • guybe7's avatar
      Obliterate STRALGO! add LCS (which only works on keys) (#9799) · af748988
      guybe7 authored
      Drop the STRALGO command, now LCS is a command of its own and it only works on keys (not input strings).
      The motivation is that STRALGO's syntax was really messed-up...
      - assumes all (future) string algorithms will take similar arguments
      - mixes command that takes keys and one that doesn't in the same command.
      - make it nearly impossible to expose the right key spec in COMMAND INFO (issues cluster clients)
      - hard for cluster clients to determine the key names (firstkey, lastkey, etc)
      - hard for ACL / flags (is it a read command?)
      
      This is a breaking change.
      af748988
    • Binbin's avatar
      Fixes ZPOPMIN/ZPOPMAX wrong replies when count is 0 with non-zset (#9711) · 91e77a0c
      Binbin authored
      Moves ZPOP ... 0 fast exit path after type check to reply with
      WRONGTYPE. In the past it will return an empty array.
      
      Also now count is not allowed to be negative.
      
      see #9680
      
      before:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (empty array)
      127.0.0.1:6379> zpopmin zset -1
      (empty array)
      ```
      
      after:
      ```
      127.0.0.1:6379> set zset str
      OK
      127.0.0.1:6379> zpopmin zset 0
      (error) WRONGTYPE Operation against a key holding the wrong kind of value
      127.0.0.1:6379> zpopmin zset -1
      (error) ERR value is out of range, must be positive
      ```
      91e77a0c
    • Madelyn Olson's avatar
      Unpause clients earlier during manual cluster failover (#9676) · 32215e78
      Madelyn Olson authored
      Unpause clients after manual failover ends instead of the timed offset
      32215e78
  8. 16 Nov, 2021 6 commits
    • Oran Agra's avatar
      Daily tests flags inconsistency (#9792) · 45144fc7
      Oran Agra authored
      Add --accurate to unit tests (new feature recently added)
      Add --no-latency to valgrind run (was present only for modules)
      add --no-latency to macos and freebsd runs (was not present for modules)
      add --timeout to freebsd (same one we have for valgrind)
      45144fc7
    • Axlgrep's avatar
      optimze the judgment logic of use_pattern in scanGenericCommand() (#9789) · 5f8ee381
      Axlgrep authored
      in `scan 0 match ""` case,  pat is empty sds(patlen is 0), I don't think should access the
      first character directly in this case(even though the first character is ' \0 '), for the
      code readability, I switch the two positions of judgment logic. 
      5f8ee381
    • harleyliao's avatar
      set aof rewrite status err, when fork fail (#5606) · 992a610f
      harleyliao authored
      when aof rewrite is failed by fork(), It'll be indicated by aof_last_bgrewrite_status
      INFO field, same as when the fork child fails later on.
      992a610f
    • sundb's avatar
      Change lzf to handle values larger than UINT32_MAX (#9776) · 985430b4
      sundb authored
      Redis supports inserting data over 4GB into string (and recently for lists too, see #9357),
      But LZF compression used in RDB files (see `rdbcompression` config), and in quicklist
      (see `list-compress-depth` config) does not support compress/decompress data over
      UINT32_MAX, which will result in corrupting the rdb after compression.
      
      Internal changes:
      1. Modify the `unsigned int` parameter of `lzf_compress/lzf_decompress` to `size_t`.
      2. Modify the variable types in `lzf_compress` involving offsets and lengths to `size_t`.
      3. Set LZF_USE_OFFSETS to 0.
          When LZF_USE_OFFSETS is 1, lzf store offset into `LZF_HSLOT`(32bit). 
          Even in 64-bit, `LZF_USE_OFFSETS` defaults to 1, because lzf assumes that it only
          compresses and decompresses data smaller than UINT32_MAX.
          But now we need to make lzf support 64-bit, turning on `LZF_USE_OFFSETS` will make
          it impossible to store 64-bit offsets or pointers.
          BTW, disable LZF_USE_OFFS...
      985430b4
    • sundb's avatar
      Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) · e725d737
      sundb authored
      This is a preparation step in order to add a new test in quicklist.c see #9776
      e725d737
    • guoxiang1996's avatar
      insufficient size for cached client flags in call() (#9783) · aba70df4
      guoxiang1996 authored
      The client flags is a 64 bit integer, but the temporary cached value on the stack of call() is 32 bit.
      luckily this doesn't lead to any bugs since the only flags used against this variables are below 32 bit.
      aba70df4
  9. 15 Nov, 2021 1 commit
    • yoav-steinberg's avatar
      Connection leak in external tests. (#9777) · e968d9ac
      yoav-steinberg authored
      Two issues:
      1. In many tests we simply forgot to close the connections we created, which doesn't matter for normal tests where the server is killed, but creates a leak on external server tests.
      2. When calling `start_server` on external test we create a fresh connection instead of really starting a new server, but never clean it at the end.
      e968d9ac
  10. 13 Nov, 2021 1 commit
    • Binbin's avatar
      Tune expire test threshold. (#9775) · 174eedce
      Binbin authored
      I have seen this CI failure twice on MacOS:
      
      *** [err]: PEXPIRE/PSETEX/PEXPIREAT can set sub-second expires in tests/unit/expire.tcl
      Expected 'somevalue {} somevalue {} somevalue {}' to equal or match '{} {} {} {} somevalue {}'
      
      I did some loop test in my own daily CI, the results show that is
      not particularly stable. Change the threshold from 30 to 50.
      174eedce
  11. 11 Nov, 2021 4 commits
    • Ozan Tezcan's avatar
      Enable running daily CI from forks (#9771) · bcb7961f
      Ozan Tezcan authored
      Was impossible to run the daily CI from private forks due to "redis/redis" repo check.
      Let's disable that check for manual triggers.
      bcb7961f
    • Ozan Tezcan's avatar
      Add sanitizer support and clean up sanitizer findings (#9601) · b91d8b28
      Ozan Tezcan authored
      - Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
      - To build Redis with desired sanitizer : `make SANITIZER=undefined`
      - There were some sanitizer findings, cleaned up codebase
      - Added tests with address and undefined behavior sanitizers to daily CI.
      - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).
      
      Basically, there are three types of issues : 
      
      **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
      does not support unaligned access. Redis does unaligned access only on supported platforms.
      
      **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
       **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit. 
      b91d8b28
    • yoav-steinberg's avatar
      Archive external redis log in external tests (#9765) · cd6b3d55
      yoav-steinberg authored
      On test failure store the external redis server logs as CI artifacts so we can review them.
      
      Write test name to server log for external server tests.
      This is attempted and silently failed in case external server doesn't support it.
      Note that in non-external server mode we use a more robust method of writing to the log which doesn't depend on the
      server actually running/working. This isn't possible for externl servers and required for some complex tests which are
      skipped in external mode anyway.
      
      Cleanup: remove dup code.
      cd6b3d55
    • Vadim Pushtaev's avatar
      f069d09e
  12. 10 Nov, 2021 2 commits
    • Oran Agra's avatar
      Try solving test timeout on freebsd CI (#9768) · 0927a0dd
      Oran Agra authored
      First, avoid using --accurate on the freebsd CI, we only care about
      systematic issues there due to being different platform, but not
      accuracy
      
      Secondly, when looking at the test which timed out it seems silly and
      outdated:
      - it used KEYS to attempt to trigger lazy expiry, but KEYS doesn't do
        that anymore.
      - it used some hard coded sleeps rather than waiting for things to
        happen and exiting ASAP
      0927a0dd
    • Oran Agra's avatar
      Increase test timeout in valgrind runs (#9767) · 978eadba
      Oran Agra authored
      We saw some tests sporadically time out on valgrind (namely the ones
      from #9323).
      
      Increasing valgrind timeout from 20 mins to 40 mins in CI.
      And fixing an outdated help message.
      978eadba
  13. 09 Nov, 2021 3 commits