1. 08 Sep, 2020 1 commit
    • bodong.ybd's avatar
      Tests: Some fixes for macOS · f22fa959
      bodong.ybd authored
      1) cur_test: when restart_server, "no such variable" error occurs
        ./runtest --single integration/rdb
        test {client freed during loading}
            SET ::cur_test
            restart_server
              kill_server
                test "Check for memory leaks (pid $pid)"
                SET ::cur_test
                UNSET ::cur_test
            UNSET ::cur_test // This global variable has been unset.
      
      2) `ps --ppid` not available on macOS platform, can be replaced with
      `pgrep -P pid`.
      f22fa959
  2. 07 Sep, 2020 1 commit
    • Yossi Gottlieb's avatar
      Tests: fix unmonitored servers. (#7756) · 2df4cb93
      Yossi Gottlieb authored
      There is an inherent race condition in port allocation for spawned
      servers. If a server fails to start because a port is taken, a new port
      is allocated. This fixes a problem where the logs are not truncated and
      as a result a large number of unmonitored servers are started.
      2df4cb93
  3. 06 Sep, 2020 8 commits
    • Oran Agra's avatar
      if diskless repl child is killed, make sure to reap the pid (#7742) · 573246f7
      Oran Agra authored
      Starting redis 6.0 and the changes we made to the diskless master to be
      suitable for TLS, I made the master avoid reaping (wait3) the pid of the
      child until we know all replicas are done reading their rdb.
      
      I did that in order to avoid a state where the rdb_child_pid is -1 but
      we don't yet want to start another fork (still busy serving that data to
      replicas).
      
      It turns out that the solution used so far was problematic in case the
      fork child was being killed (e.g. by the kernel OOM killer), in that
      case there's a chance that we currently disabled the read event on the
      rdb pipe, since we're waiting for a replica to become writable again.
      and in that scenario the master would have never realized the child
      exited, and the replica will remain hung too.
      Note that there's no mechanism to detect a hung replica while it's in
      rdb transfer state.
      
      The solution here is to add another pipe which is used by the parent to
      tell the child it is safe to exit. this mean that when the child exits,
      for whatever reason, it is safe to reap it.
      
      Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was
      part of #6271 (Accelerate diskless master connections) but was dropped
      when that PR was rebased after the TLS fork/pipe changes (5a477946).
      Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK
      has chance to detect that the child exited, it should be the one to call
      it so that we don't have to wait for cron (server.hz) to do that.
      573246f7
    • Oran Agra's avatar
      Improve valgrind support for cluster tests (#7725) · 2b998de4
      Oran Agra authored
      - redirect valgrind reports to a dedicated file rather than console
      - try to avoid killing instances with SIGKILL so that we get the memory
        leak report (killing with SIGTERM before resorting to SIGKILL)
      - search for valgrind reports when done, print them and fail the tests
      - add --dont-clean option to keep the logs on exit
      - fix exit error code when crash is found (would have exited with 0)
      
      changes that affect the normal redis test suite:
      - refactor check_valgrind_errors into two functions one to search and
        one to report
      - move the search half into util.tcl to serve the cluster tests too
      - ignore "address range perms" valgrind warnings which seem non relevant.
      2b998de4
    • Oran Agra's avatar
      test infra - add durable mode to work around test suite crashing · fe5da2e6
      Oran Agra authored
      in some cases a command that returns an error possibly due to a timing
      issue causes the tcl code to crash and thus prevents the rest of the
      tests from running. this adds an option to make the test proceed despite
      the crash.
      maybe it should be the default mode some day.
      fe5da2e6
    • Oran Agra's avatar
      test infra - wait_done_loading · 1b7ba44e
      Oran Agra authored
      reduce code duplication in aof.tcl.
      move creation of clients into the test so that it can be skipped
      1b7ba44e
    • Oran Agra's avatar
      b65e5aca
    • Oran Agra's avatar
      test infra - improve test skipping ability · 677d14c2
      Oran Agra authored
      - skip full units
      - skip a single test (not just a list of tests)
      - when skipping tag, skip spinning up servers, not just the tests
      - skip tags when running against an external server too
      - allow using multiple tags (split them)
      677d14c2
    • Oran Agra's avatar
      test infra - reduce disk space usage · e3e69c25
      Oran Agra authored
      this is important when running a test with --loop
      e3e69c25
    • Oran Agra's avatar
      test infra - write test name to logfile · 9d527d07
      Oran Agra authored
      9d527d07
  4. 03 Sep, 2020 1 commit
    • Oran Agra's avatar
      Run active defrag while blocked / loading (#7726) · 9ef8d2f6
      Oran Agra authored
      During long running scripts or loading RDB/AOF, we may need to do some
      defragging. Since processEventsWhileBlocked is called periodically at
      unknown intervals, and many cron jobs either depend on run_with_period
      (including active defrag), or rely on being called at server.hz rate
      (i.e. active defrag knows ho much time to run by looking at server.hz),
      the whileBlockedCron may have to run a loop triggering the cron jobs in it
      (currently only active defrag) several times.
      
      Other changes:
      - Adding a test for defrag during aof loading.
      - Changing key-load-delay config to take negative values for fractions
        of a microsecond sleep
      9ef8d2f6
  5. 23 Aug, 2020 1 commit
  6. 11 Aug, 2020 1 commit
    • Tyson Andre's avatar
      Implement SMISMEMBER key member [member ...] (#7615) · 6f11acbd
      Tyson Andre authored
      
      
      This is a rebased version of #3078 originally by shaharmor
      with the following patches by TysonAndre made after rebasing
      to work with the updated C API:
      
      1. Add 2 more unit tests
         (wrong argument count error message, integer over 64 bits)
      2. Use addReplyArrayLen instead of addReplyMultiBulkLen.
      3. Undo changes to src/help.h - for the ZMSCORE PR,
         I heard those should instead be automatically
         generated from the redis-doc repo if it gets updated
      
      Motivations:
      
      - Example use case: Client code to efficiently check if each element of a set
        of 1000 items is a member of a set of 10 million items.
        (Similar to reasons for working on #7593)
      - HMGET and ZMSCORE already exist. This may lead to developers deciding
        to implement functionality that's best suited to a regular set with a
        data type of sorted set or hash map instead, for the multi-get support.
      
      Currently, multi commands or lua scripting to call sismember multiple times
      would almost definitely be less efficient than a native smismember
      for the following reasons:
      
      - Need to fetch the set from the string every time
        instead of reusing the C pointer.
      - Using pipelining or multi-commands would result in more bytes sent
        and received by the client for the repeated SISMEMBER KEY sections.
      - Need to specially encode the data and decode it from the client
        for lua-based solutions.
      - Proposed solutions using Lua or SADD/SDIFF could trigger writes to
        memory, which is undesirable on a redis replica server
        or when commands get replicated to replicas.
      Co-Authored-By: default avatarShahar Mor <shahar@peer5.com>
      Co-Authored-By: default avatarTyson Andre <tysonandre775@hotmail.com>
      6f11acbd
  7. 04 Aug, 2020 1 commit
    • Tyson Andre's avatar
      Add a ZMSCORE command returning an array of scores. (#7593) · f11f26cc
      Tyson Andre authored
      
      
      Syntax: `ZMSCORE KEY MEMBER [MEMBER ...]`
      
      This is an extension of #2359
      amended by Tyson Andre to work with the changed unstable API,
      add more tests, and consistently return an array.
      
      - It seemed as if it would be more likely to get reviewed
        after updating the implementation.
      
      Currently, multi commands or lua scripting to call zscore multiple times
      would almost definitely be less efficient than a native ZMSCORE
      for the following reasons:
      
      - Need to fetch the set from the string every time instead of reusing the C
        pointer.
      - Using pipelining or multi-commands would result in more bytes sent by
        the client for the repeated `ZMSCORE KEY` sections.
      - Need to specially encode the data and decode it from the client
        for lua-based solutions.
      - The fastest solution I've seen for large sets(thousands or millions)
        involves lua and a variadic ZADD, then a ZINTERSECT, then a ZRANGE 0 -1,
        then UNLINK of a temporary set (or lua). This is still inefficient.
      Co-authored-by: default avatarTyson Andre <tysonandre775@hotmail.com>
      f11f26cc
  8. 28 Jul, 2020 1 commit
    • Oran Agra's avatar
      Fix failing tests due to issues with wait_for_log_message (#7572) · 109b5ccd
      Oran Agra authored
      - the test now waits for specific set of log messages rather than wait for
        timeout looking for just one message.
      - we don't wanna sample the current length of the log after an action, due
        to a race, we need to start the search from the line number of the last
        message we where waiting for.
      - when attempting to trigger a full sync, use multi-exec to avoid a race
        where the replica manages to re-connect before we completed the set of
        actions that should force a full sync.
      - fix verify_log_message which was broken and unused
      109b5ccd
  9. 21 Jul, 2020 2 commits
    • Oran Agra's avatar
      testsuite may leave servers alive on error (#7549) · 36b94943
      Oran Agra authored
      in cases where you have
      test name {
        start_server {
          start_server {
            assert
          }
        }
      }
      
      the exception will be thrown to the test proc, and the servers are
      supposed to be killed on the way out. but it seems there was always a
      bug of not cleaning the server stack, and recently (#7404) we started
      relying on that stack in order to kill them, so with that bug sometimes
      we would have tried to kill the same server twice, and leave one alive.
      
      luckly, in most cases the pattern is:
      start_server {
        test name {
        }
      }
      36b94943
    • Remi Collet's avatar
      Fix deprecated tail syntax in tests (#7543) · 3f2fbc4c
      Remi Collet authored
      3f2fbc4c
  10. 13 Jul, 2020 1 commit
    • Oran Agra's avatar
      runtest --stop pause stops before terminating the redis server (#7513) · 02ef355f
      Oran Agra authored
      in the majority of the cases (on this rarely used feature) we want to
      stop and be able to connect to the shard with redis-cli.
      since these are two different processes interracting with the tty we
      need to stop both, and we'll have to hit enter twice, but it's not that
      bad considering it is rarely used.
      02ef355f
  11. 10 Jul, 2020 2 commits
    • Oran Agra's avatar
      stabilize tests that look for log lines (#7367) · 8e76e134
      Oran Agra authored
      tests were sensitive to additional log lines appearing in the log
      causing the search to come empty handed.
      
      instead of just looking for the n last log lines, capture the log lines
      before performing the action, and then search from that offset.
      8e76e134
    • Oran Agra's avatar
      tests/valgrind: don't use debug restart (#7404) · 69ade873
      Oran Agra authored
      * tests/valgrind: don't use debug restart
      
      DEBUG REATART causes two issues:
      1. it uses execve which replaces the original process and valgrind doesn't
         have a chance to check for errors, so leaks go unreported.
      2. valgrind report invalid calls to close() which we're unable to resolve.
      
      So now the tests use restart_server mechanism in the tests, that terminates
      the old server and starts a new one, new PID, but same stdout, stderr.
      
      since the stderr can contain two or more valgrind report, it is not enough
      to just check for the absence of leaks, we also need to check for some known
      errors, we do both, and fail if we either find an error, or can't find a
      report saying there are no leaks.
      
      other changes:
      - when killing a server that was already terminated we check for leaks too.
      - adding DEBUG LEAK which was used to test it.
      - adding --trace-children to valgrind, although no longer needed.
      - since the stdout contains two or more runs, we need slightly different way
        of checking if the new process is up (explicitly looking for the new PID)
      - move the code that handles --wait-server to happen earlier (before
        watching the startup message in the log), and serve the restarted server too.
      
      * squashme - CR fixes
      69ade873
  12. 27 May, 2020 1 commit
    • Oran Agra's avatar
      tests: find_available_port start search from next port · 1cf33a46
      Oran Agra authored
      i.e. don't start the search from scratch hitting the used ones again.
      this will also reduce the likelihood of collisions (if there are any
      left) by increasing the time until we re-use a port we did use in the
      past.
      1cf33a46
  13. 26 May, 2020 1 commit
    • Oran Agra's avatar
      tests: each test client work on a distinct port range · e258a1c0
      Oran Agra authored
      apparently when running tests in parallel (the default of --clients 16),
      there's a chance for two tests to use the same port.
      specifically, one test might shutdown a master and still have the
      replica up, and then another test will re-use the port number of master
      for another master, and then that replica will connect to the master of
      the other test.
      
      this can cause a master to count too many full syncs and fail a test if
      we run the tests with --single integration/psync2 --loop --stop
      
      see Probmem 2 in #7314
      e258a1c0
  14. 11 May, 2020 1 commit
  15. 06 Apr, 2020 1 commit
  16. 11 Mar, 2020 1 commit
  17. 24 Feb, 2020 1 commit
  18. 21 Feb, 2020 2 commits
  19. 29 Oct, 2019 1 commit
    • Oran Agra's avatar
      test infra: improve prints on failed assertions · a0cfd519
      Oran Agra authored
      sometimes we have several assertions with the same condition in the same test
      at different stages, and when these fail (the ones that print the condition
      text) you don't know which one it was. other assertions didn't print the
      condition text (variable names), just the expected and unexpected values.
      
      So now, all assertions print context line, and conditin text.
      
      besides, one of the major differences between 'assert' and 'assert_equal',
      is that the later is able to print the value that doesn't match the expected.
      if there is a rare non-reproducible failure, it is helpful to know what was
      the value the test encountered and how far it was from the threshold.
      
      So now, adding assert_lessthan and assert_range that can be used in some places.
      were we used just 'assert { a > b }' so far.
      a0cfd519
  20. 07 Oct, 2019 2 commits
    • Yossi Gottlieb's avatar
      TLS: Configuration options. · 61733ded
      Yossi Gottlieb authored
      Add configuration options for TLS protocol versions, ciphers/cipher
      suites selection, etc.
      61733ded
    • Yossi Gottlieb's avatar
      TLS: Connections refactoring and TLS support. · b087dd1d
      Yossi Gottlieb authored
      * Introduce a connection abstraction layer for all socket operations and
      integrate it across the code base.
      * Provide an optional TLS connections implementation based on OpenSSL.
      * Pull a newer version of hiredis with TLS support.
      * Tests, redis-cli updates for TLS support.
      b087dd1d
  21. 30 Sep, 2019 1 commit
  22. 17 Jul, 2019 1 commit
    • Oran Agra's avatar
      prevent diskless replica from terminating on short read · c56b4ddc
      Oran Agra authored
      now that replica can read rdb directly from the socket, it should avoid exiting
      on short read and instead try to re-sync.
      
      this commit tries to have minimal effects on non-diskless rdb reading.
      and includes a test that tries to trigger this scenario on various read cases.
      c56b4ddc
  23. 08 Jul, 2019 1 commit
    • Oran Agra's avatar
      diskless replication on slave side (don't store rdb to file), plus some other related fixes · 2de544cf
      Oran Agra authored
      The implementation of the diskless replication was currently diskless only on the master side.
      The slave side was still storing the received rdb file to the disk before loading it back in and parsing it.
      
      This commit adds two modes to load rdb directly from socket:
      1) when-empty
      2) using "swapdb"
      the third mode of using diskless slave by flushdb is risky and currently not included.
      
      other changes:
      --------------
      distinguish between aof configuration and state so that we can re-enable aof only when sync eventually
      succeeds (and not when exiting from readSyncBulkPayload after a failed attempt)
      also a CONFIG GET and INFO during rdb loading would have lied
      
      When loading rdb from the network, don't kill the server on short read (that can be a network error)
      
      Fix rdb check when performed on preamble AOF
      
      tests:
      run replication tests for diskless slave too
      make replication test a bit more aggressive
      Add test for diskless load swapdb
      2de544cf
  24. 12 Nov, 2018 1 commit
  25. 04 Sep, 2018 1 commit
  26. 30 Jul, 2018 1 commit
    • Oran Agra's avatar
      test suite conveniency improvements · 78292876
      Oran Agra authored
      * allowing --single to be repeated
      * adding --only so that only a specific test inside a unit can be run
      * adding --skiptill useful to resume a test that crashed passed the problematic unit.
        useful together with --clients 1
      * adding --skipfile to use a file containing list of tests names to skip
      * printing the names of the tests that are skiped by skipfile or denytags
      * adding --config to add config file options from command line
      78292876
  27. 13 Jul, 2018 2 commits
  28. 26 Jun, 2018 1 commit
    • Oran Agra's avatar
      test suite infra improvements and fix · 751eea24
      Oran Agra authored
      * fail the test (exit code) in case of timeout.
      * add --wait-server to allow attaching a debugger
      * add --dont-clean to keep log files when tests are done
      751eea24