1. 16 Nov, 2021 2 commits
  2. 15 Nov, 2021 1 commit
    • yoav-steinberg's avatar
      Connection leak in external tests. (#9777) · e968d9ac
      yoav-steinberg authored
      Two issues:
      1. In many tests we simply forgot to close the connections we created, which doesn't matter for normal tests where the server is killed, but creates a leak on external server tests.
      2. When calling `start_server` on external test we create a fresh connection instead of really starting a new server, but never clean it at the end.
      e968d9ac
  3. 13 Nov, 2021 1 commit
    • Binbin's avatar
      Tune expire test threshold. (#9775) · 174eedce
      Binbin authored
      I have seen this CI failure twice on MacOS:
      
      *** [err]: PEXPIRE/PSETEX/PEXPIREAT can set sub-second expires in tests/unit/expire.tcl
      Expected 'somevalue {} somevalue {} somevalue {}' to equal or match '{} {} {} {} somevalue {}'
      
      I did some loop test in my own daily CI, the results show that is
      not particularly stable. Change the threshold from 30 to 50.
      174eedce
  4. 11 Nov, 2021 4 commits
    • Ozan Tezcan's avatar
      Enable running daily CI from forks (#9771) · bcb7961f
      Ozan Tezcan authored
      Was impossible to run the daily CI from private forks due to "redis/redis" repo check.
      Let's disable that check for manual triggers.
      bcb7961f
    • Ozan Tezcan's avatar
      Add sanitizer support and clean up sanitizer findings (#9601) · b91d8b28
      Ozan Tezcan authored
      - Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
      - To build Redis with desired sanitizer : `make SANITIZER=undefined`
      - There were some sanitizer findings, cleaned up codebase
      - Added tests with address and undefined behavior sanitizers to daily CI.
      - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).
      
      Basically, there are three types of issues : 
      
      **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
      does not support unaligned access. Redis does unaligned access only on supported platforms.
      
      **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
       **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit. 
      b91d8b28
    • yoav-steinberg's avatar
      Archive external redis log in external tests (#9765) · cd6b3d55
      yoav-steinberg authored
      On test failure store the external redis server logs as CI artifacts so we can review them.
      
      Write test name to server log for external server tests.
      This is attempted and silently failed in case external server doesn't support it.
      Note that in non-external server mode we use a more robust method of writing to the log which doesn't depend on the
      server actually running/working. This isn't possible for externl servers and required for some complex tests which are
      skipped in external mode anyway.
      
      Cleanup: remove dup code.
      cd6b3d55
    • Vadim Pushtaev's avatar
      f069d09e
  5. 10 Nov, 2021 2 commits
    • Oran Agra's avatar
      Try solving test timeout on freebsd CI (#9768) · 0927a0dd
      Oran Agra authored
      First, avoid using --accurate on the freebsd CI, we only care about
      systematic issues there due to being different platform, but not
      accuracy
      
      Secondly, when looking at the test which timed out it seems silly and
      outdated:
      - it used KEYS to attempt to trigger lazy expiry, but KEYS doesn't do
        that anymore.
      - it used some hard coded sleeps rather than waiting for things to
        happen and exiting ASAP
      0927a0dd
    • Oran Agra's avatar
      Increase test timeout in valgrind runs (#9767) · 978eadba
      Oran Agra authored
      We saw some tests sporadically time out on valgrind (namely the ones
      from #9323).
      
      Increasing valgrind timeout from 20 mins to 40 mins in CI.
      And fixing an outdated help message.
      978eadba
  6. 09 Nov, 2021 3 commits
  7. 08 Nov, 2021 3 commits
  8. 07 Nov, 2021 2 commits
    • chendianqiang's avatar
      Test suite - user server socket to optimize port detection (#9663) · a527c3e8
      chendianqiang authored
      
      
      Optimized port detection for tcl, use 'socket -server' instead of 'socket' to rule out port on TIME_WAIT
      Co-authored-by: default avatarchendianqiang <chendianqiang@meituan.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      a527c3e8
    • yoav-steinberg's avatar
      Refactor config.c for generic setter interface (#9644) · 79ac5756
      yoav-steinberg authored
      
      
      This refactors all `CONFIG SET`s and conf file loading arguments go through
      the generic config handling interface.
      
      Refactoring changes:
      - All config params go through the `standardConfig` interface (some stuff which
        is only related to the config file and not the `CONFIG` command still has special
        handling for rewrite/config file parsing, `loadmodule`, for example.) .
      - Added `MULTI_ARG_CONFIG` flag for configs to signify they receive a variable
        number of arguments instead of a single argument. This is used to break up space
        separated arguments to `CONFIG SET` so the generic setter interface can pass
        multiple arguments to the setter function. When parsing the config file we also break
        up anything after the config name into multiple arguments to the setter function.
      
      Interface changes:
      - A side effect of the above interface is that the `bind` argument in the config file can
        be empty (no argument at all) this is treated the same as passing an single empty
        string argument (same as `save` already used to work).
      - Support rewrite and setting `watchdog-period` from config file (was only supported
        by the CONFIG command till now).
      - Another side effect is that the `save T X` config argument now supports multiple
        Time-Changes pairs in a single line like its `CONFIG SET` counterpart. So in the
        config file you can either do:
        ```
        save 3600 1
        save 600 10
        ```
        or do
        ```
        save 3600 1 600 10
        ```
      Co-authored-by: default avatarBjorn Svensson <bjorn.a.svensson@est.tech>
      79ac5756
  9. 05 Nov, 2021 1 commit
  10. 04 Nov, 2021 4 commits
    • Eduardo Semprebon's avatar
      Replica keep serving data during repl-diskless-load=swapdb for better availability (#9323) · 91d0c758
      Eduardo Semprebon authored
      
      
      For diskless replication in swapdb mode, considering we already spend replica memory
      having a backup of current db to restore in case of failure, we can have the following benefits
      by instead swapping database only in case we succeeded in transferring db from master:
      
      - Avoid `LOADING` response during failed and successful synchronization for cases where the
        replica is already up and running with data.
      - Faster total time of diskless replication, because now we're moving from Transfer + Flush + Load
        time to Transfer + Load only. Flushing the tempDb is done asynchronously after swapping.
      - This could be implemented also for disk replication with similar benefits if consumers are willing
        to spend the extra memory usage.
      
      General notes:
      - The concept of `backupDb` becomes `tempDb` for clarity.
      - Async loading mode will only kick in if the replica is syncing from a master that has the same
        repl-id the one it had before. i.e. the data it's getting belongs to a different time of the same timeline. 
      - New property in INFO: `async_loading` to differentiate from the blocking loading
      - Slot to Key mapping is now a field of `redisDb` as it's more natural to access it from both server.db
        and the tempDb that is passed around.
      - Because this is affecting replicas only, we assume that if they are not readonly and write commands
        during replication, they are lost after SYNC same way as before, but we're still denying CONFIG SET
        here anyways to avoid complications.
      
      Considerations for review:
      - We have many cases where server.loading flag is used and even though I tried my best, there may
        be cases where async_loading should be checked as well and cases where it shouldn't (would require
        very good understanding of whole code)
      - Several places that had different behavior depending on the loading flag where actually meant to just
        handle commands coming from the AOF client differently than ones coming from real clients, changed
        to check CLIENT_ID_AOF instead.
      
      **Additional for Release Notes**
      - Bugfix - server.dirty was not incremented for any kind of diskless replication, as effect it wouldn't
        contribute on triggering next database SAVE
      - New flag for RM_GetContextFlags module API: REDISMODULE_CTX_FLAGS_ASYNC_LOADING
      - Deprecated RedisModuleEvent_ReplBackup. Starting from Redis 7.0, we don't fire this event.
        Instead, we have the new RedisModuleEvent_ReplAsyncLoad holding 3 sub-events: STARTED,
        ABORTED and COMPLETED.
      - New module flag REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD for RedisModule_SetModuleOptions
        to allow modules to declare they support the diskless replication with async loading (when absent, we fall
        back to disk-based loading).
      Co-authored-by: default avatarEduardo Semprebon <edus@saxobank.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      91d0c758
    • Itamar Haber's avatar
      Fixes LPOP/RPOP wrong replies when count is 0 (#9692) · 06dd202a
      Itamar Haber authored
      Introduced in #8179, this fixes the command's replies in the 0 count edge case.
      [BREAKING] changes the reply type when count is 0 to an empty array (instead of nil)
      Moves LPOP ... 0 fast exit path after type check to reply with WRONGTYPE
      06dd202a
    • menwen's avatar
      Retry when a blocked connection system call is interrupted by a signal (#9629) · ccf8a651
      menwen authored
      
      
      When repl-diskless-load is enabled, the connection is set to the blocking state.
      The connection may be interrupted by a signal during a system call.
      This would have resulted in a disconnection and possibly a reconnection loop.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ccf8a651
    • Oran Agra's avatar
      Fix race condition in cluster test 22-replica-in-sync (#9721) · d04f3069
      Oran Agra authored
      
      
      there was a chance that by the time the assertion is executed,
      the replica already manages to reconnect.
      
      now we make sure the replica is unable to re-connect to the master.
      
      additionally, we wait for some gossip from the disconnected replica,
      to see that it doesn't mess things up.
      
      unrelated: fix a typo when trying to exhaust the backlog, one that
      didn't have any harmful implications
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      d04f3069
  11. 03 Nov, 2021 3 commits
    • perryitay's avatar
      Add support for list type to store elements larger than 4GB (#9357) · f27083a4
      perryitay authored
      
      
      Redis lists are stored in quicklist, which is currently a linked list of ziplists.
      Ziplists are limited to storing elements no larger than 4GB, so when bigger
      items are added they're getting truncated.
      This PR changes quicklists so that they're capable of storing large items
      in quicklist nodes that are plain string buffers rather than ziplist.
      
      As part of the PR there were few other changes in redis: 
      1. new DEBUG sub-commands: 
         - QUICKLIST-PACKED-THRESHOLD - set the threshold of for the node type to
           be plan or ziplist. default (1GB)
         - QUICKLIST <key> - Shows low level info about the quicklist encoding of <key>
      2. rdb format change:
         - A new type was added - RDB_TYPE_LIST_QUICKLIST_2 . 
         - container type (packed / plain) was added to the beginning of the rdb object
           (before the actual node list).
      3. testing:
         - Tests that requires over 100MB will be by default skipped. a new flag was
           added to 'runtest' to run the large memory tests (not used by default)
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      f27083a4
    • guybe7's avatar
      Fix COMMAND GETKEYS on EVAL without keys (#9733) · f11a2d4d
      guybe7 authored
      Add new no-mandatory-keys flag to support COMMAND GETKEYS of commands
      which have no mandatory keys.
      
      In the past we would have got this error:
      ```
      127.0.0.1:6379> command getkeys eval "return 1" 0
      (error) ERR Invalid arguments specified for command
      ```
      f11a2d4d
    • perryitay's avatar
      fix: lookupKey on SETNX and SETXX only once (#9640) · 77d3c6bf
      perryitay authored
      When using SETNX and SETXX we could end up doing key lookup twice.
      This presents a small inefficiency price.
      Also once we have statistics of write hit and miss they'll be wrong (recording the same key hit twice) 
      77d3c6bf
  12. 02 Nov, 2021 7 commits
    • Oran Agra's avatar
      Solve issues with tracking test in external mode (#9726) · d25dc089
      Oran Agra authored
      The issue was that setting maxmemory to used_memory and expecting
      eviction is insufficient, since we need to take
      mem_not_counted_for_evict into consideration.
      
      This test got broken by #9166
      d25dc089
    • yiyuaner's avatar
    • Oran Agra's avatar
      attempt to fix tracking test issue with external tests due to lazy free (#9722) · 87321deb
      Oran Agra authored
      The External tests started failing recently for unclear reason:
      ```
      *** [err]: Tracking invalidation message of eviction keys should be before response in tests/unit/tracking.tcl
      Expected '0' to be equal to 'invalidate volatile-key' (context: type eval line 21 cmd {assert_equal $res {invalidate volatile-key}} proc ::test)
      ```
      
      I suspect the issue is that the used_memory sample is taken while a lazy free is still being processed.
      87321deb
    • menwen's avatar
      fix defrag test looking at the wrong latency metric (#9723) · d5ca72e3
      menwen authored
      the latency event was renamed in #7726, and the outcome was that the test was
      ineffective (unable to measure the max latency, always seeing 0)
      d5ca72e3
    • Wang Yuan's avatar
      Fix not updating backlog histlen when trimming repl backlog (#9713) · 526cbb5c
      Wang Yuan authored
      
      
      Since the loop in incrementalTrimReplicationBacklog checks the size of histlen,
      we cannot afford to update it only when the loop exits, this may cause deleting
      much more replication blocks, and replication backlog may be less than setting size.
      
      introduce in #9166 
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      526cbb5c
    • zhaozhao.zz's avatar
      rebuild replication backlog index when master restart (#9720) · d08f0552
      zhaozhao.zz authored
      After PR #9166 , replication backlog is not a real block of memory, just contains a
      reference points to replication buffer's block and the blocks index (to accelerate
      search offset when partial sync), so we need update both replication buffer's block's
      offset and replication backlog blocks index's offset when master restart from RDB,
      since the `server.master_repl_offset` is changed.
      The implications of this bug was just a slow search, but not a replication failure.
      d08f0552
    • Binbin's avatar
      Fix timing issue in replication test (#9719) · 58a1d16f
      Binbin authored
      
      
      So it looks like sampling set loglines [count_log_lines -2] was
      executed too late, and the replication managed to complete before that.
      
      ```
      *** [err]: diskless no replicas drop during rdb pipe in tests/integration/replication.tcl
      log message of '"*Diskless rdb transfer, done reading from pipe, 2 replicas still up*"' not found in ./tests/tmp/server.6124.69/stdout after line: 52 till line: 52
      ```
      
      Changes:
      1. when we search the master log file, we start to search from before we sent the REPLICAOF
        command, to prevent a race in which the replication completed before we sampled the log line count.
      2. we don't need to sample the replica loglines sine it's a fresh resplica that's just been started, so the message
        we're looking for is the first occurrence in the log, we can start search from 0.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      58a1d16f
  13. 01 Nov, 2021 3 commits
    • Binbin's avatar
      Fix race condition in psync2-pingoff test (#9712) · cea7809c
      Binbin authored
      
      
      Test failed on freebsd:
      ```
      *** [err]: Make the old master a replica of the new one and check conditions in tests/integration/psync2-pingoff.tcl
      Expected '162' to be equal to '176' (context: type eval line 18 cmd {assert_equal [status $R(0) master_repl_offset] [status $R(1) master_repl_offset]} proc ::test)
      ```
      
      There are two possible race conditions in the test.
      
      1. The code waits for sync_full to increment, and assumes that means the
      master did the fork. But in fact there are cases the master will increment
      that sync_full counter (after replica asks for sync), but will see that
      there's already a fork running and will delay the fork creation.
      
      In this case the INCR will be executed before the fork happens, so it'll
      not be in the command stream. Solve that by waiting for `master_link_status: up`
      on the replica before the INCR.
      
      2. The repl-ping-replica-period is still high (1 second), so there's a chance the
      master will send an additional PING between the two calls to INFO (the line that
      fails is the one that samples INFO from both servers). So there's a chance one of
      them will have an incremented offset due to PING and the other won't have it yet.
      
      In theory we can wait for the repl_offset to match, but then we risk facing a
      situation where that race will hide an offset mis-match. so instead, i think we
      should just change repl-ping-replica-period to prevent further pings from being pushed.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      cea7809c
    • Oran Agra's avatar
      fix valgrind issues with long double module test (#9709) · f1f3cceb
      Oran Agra authored
      The module test in reply.tcl was introduced by #8521 but didn't run until recently (see #9639)
      and then it started failing with valgrind.
      This is because valgrind uses 64 bit long double (unlike most other platforms that have at least 80 bits)
      But besides valgrind, the tests where also incompatible with ARM32, which also uses 64 bit long doubles.
      
      We now use appropriate value to avoid issues with either valgrind or ARM32
      
      In all the double tests, i use 3.141, which is safe since since addReplyDouble uses
      `%.17Lg` which is able to represent this value without adding any digits due to precision loss. 
      
      In the long double, since we use `%.17Lf` in ld2string, it preserves 17 significant
      digits, rather than 17 digit after the decimal point (like in `%.17Lg`).
      So to make these similar, i use value lower than 1 (no digits left of
      the period)
      
      Lastly, we have the same issue with TCL (no long doubles) so we read
      raw protocol in that test.
      
      Note that the only error before this fix (in both valgrind and ARM32 is this:
      ```
      *** [err]: RM_ReplyWithLongDouble: a float reply in tests/unit/moduleapi/reply.tcl
      Expected '3.141' to be equal to '3.14100000000000001' (context: type eval line 2 cmd {assert_equal 3.141 [r rw.longdouble 3.141]} proc ::test)
      ```
      so the changes to debug.c and scripting.tcl aren't really needed, but i consider them a cleanup
      (i.e. scripting.c validated a different constant than the one that's sent to it from debug.c).
      
      Another unrelated change is to add the RESP version to the repeated tests in reply.tcl
      f1f3cceb
    • 罗泽轩's avatar
      Remove duplicate SET_OP_XX definitions in t_set.c. (#4326) · 155c2910
      罗泽轩 authored
      These definitions already exist in server.h.
      155c2910
  14. 31 Oct, 2021 4 commits
    • Oran Agra's avatar
      Fix failing cluster tests (#9707) · 48d54265
      Oran Agra authored
      Fix failures introduced by #9695 which was an attempt to solve failures introduced by #9679.
      And alternative to #9703 (i didn't like the extra argument to kill_instance).
      
      Reverting #9695.
      Instead of stopping AOF on all terminations, stop it only on the two which need it.
      Do it as part of the test rather than the infra (it was add that kill_instance used `R`
      to communicate to the instance)
      
      Note that the original purpose of these tests was to trigger a crash, but that upsets
      valgrind so in redis 6.2 i changed it to use SIGTERM, so i now rename the tests
      (remove "kill" and "crash").
      
      Also add some colors to failures, and the word "FAILED" so that it's searchable.
      
      And solve a semi-related race condition in 14-consistency-check.tcl
      48d54265
    • Yossi Gottlieb's avatar
      Use 'gcc' instead of 'ld' to link test modules. (#9710) · f26e90be
      Yossi Gottlieb authored
      This solves several problems in a more elegant way:
      
      * No need to explicitly use `-lc` on x86_64 when building with `-m32`.
      * Avoids issues with undefined floating point emulation funcs on ARM.
      f26e90be
    • Binbin's avatar
      Fix multiple COUNT in LMPOP/BLMPOP/ZMPOP/BZMPOP (#9701) · 03357883
      Binbin authored
      The previous code did not check whether COUNT is set.
      So we can use `lmpop 2 key1 key2 left count 1 count 2`.
      
      This situation can occur in LMPOP/BLMPOP/ZMPOP/BZMPOP commands.
      LMPOP/BLMPOP introduced in #9373, ZMPOP/BZMPOP introduced in #9484.
      03357883
    • lijinliang's avatar
      215b909c