1. 08 Sep, 2020 2 commits
    • Oran Agra's avatar
      handle cur_test for nested tests · 0a1e7341
      Oran Agra authored
      if there are nested tests and nested servers, we need to restore the
      previous value of cur_test when a test exist.
      
      example:
      ```
      test{test 1} {
      	start_server {
      		test{test 1.1 - master only} {
      		}
      		start_server {
      		    test{test 1.2 - with replication} {
                  }
      		}
      	}
      }
      ```
      when `test 1.1 - master only exists`, we're still inside `test 1`
      0a1e7341
    • bodong.ybd's avatar
      Tests: Some fixes for macOS · f22fa959
      bodong.ybd authored
      1) cur_test: when restart_server, "no such variable" error occurs
        ./runtest --single integration/rdb
        test {client freed during loading}
            SET ::cur_test
            restart_server
              kill_server
                test "Check for memory leaks (pid $pid)"
                SET ::cur_test
                UNSET ::cur_test
            UNSET ::cur_test // This global variable has been unset.
      
      2) `ps --ppid` not available on macOS platform, can be replaced with
      `pgrep -P pid`.
      f22fa959
  2. 07 Sep, 2020 3 commits
    • Oran Agra's avatar
      Fix cluster consistency-check test (#7754) · b491d477
      Oran Agra authored
      This test was failing from time to time see discussion at the bottom of #7635
      This was probably due to timing, the DEBUG SLEEP executed by redis-cli
      didn't sleep for enough time.
      
      This commit changes:
      1) use SET-ACTIVE-EXPIRE instead of DEBUG SLEEP
      2) reduce many `after` sleeps with retry loops to speed up the test.
      3) add many comment explaining the different steps of the test and
         it's purpose.
      4) config appendonly before populating the volatile keys, so that they'll
         be part of the AOF command stream rather than the preamble RDB portion.
      
      other complications: recently kill_instance switched from SIGKILL to
      SIGTERM, and this would sometimes fail since there was an AOFRW running
      in the background. now we wait for it to end before attempting the kill.
      b491d477
    • Yossi Gottlieb's avatar
      Tests: fix unmonitored servers. (#7756) · 2df4cb93
      Yossi Gottlieb authored
      There is an inherent race condition in port allocation for spawned
      servers. If a server fails to start because a port is taken, a new port
      is allocated. This fixes a problem where the logs are not truncated and
      as a result a large number of unmonitored servers are started.
      2df4cb93
    • Oran Agra's avatar
      fix broken cluster/sentinel tests by recent commit (#7752) · 42ba7a1b
      Oran Agra authored
      2b998de4 added a file for stderr to keep valgrind log but i forgot to
      add a similar thing when valgrind isn't being used.
      the result is that `glob */err.txt` fails.
      42ba7a1b
  3. 06 Sep, 2020 8 commits
    • Oran Agra's avatar
      if diskless repl child is killed, make sure to reap the pid (#7742) · 573246f7
      Oran Agra authored
      Starting redis 6.0 and the changes we made to the diskless master to be
      suitable for TLS, I made the master avoid reaping (wait3) the pid of the
      child until we know all replicas are done reading their rdb.
      
      I did that in order to avoid a state where the rdb_child_pid is -1 but
      we don't yet want to start another fork (still busy serving that data to
      replicas).
      
      It turns out that the solution used so far was problematic in case the
      fork child was being killed (e.g. by the kernel OOM killer), in that
      case there's a chance that we currently disabled the read event on the
      rdb pipe, since we're waiting for a replica to become writable again.
      and in that scenario the master would have never realized the child
      exited, and the replica will remain hung too.
      Note that there's no mechanism to detect a hung replica while it's in
      rdb transfer state.
      
      The solution here is to add another pipe which is used by the parent to
      tell the child it is safe to exit. this mean that when the child exits,
      for whatever reason, it is safe to reap it.
      
      Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was
      part of #6271 (Accelerate diskless master connections) but was dropped
      when that PR was rebased after the TLS fork/pipe changes (5a477946).
      Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK
      has chance to detect that the child exited, it should be the one to call
      it so that we don't have to wait for cron (server.hz) to do that.
      573246f7
    • Oran Agra's avatar
      Improve valgrind support for cluster tests (#7725) · 2b998de4
      Oran Agra authored
      - redirect valgrind reports to a dedicated file rather than console
      - try to avoid killing instances with SIGKILL so that we get the memory
        leak report (killing with SIGTERM before resorting to SIGKILL)
      - search for valgrind reports when done, print them and fail the tests
      - add --dont-clean option to keep the logs on exit
      - fix exit error code when crash is found (would have exited with 0)
      
      changes that affect the normal redis test suite:
      - refactor check_valgrind_errors into two functions one to search and
        one to report
      - move the search half into util.tcl to serve the cluster tests too
      - ignore "address range perms" valgrind warnings which seem non relevant.
      2b998de4
    • Oran Agra's avatar
      test infra - add durable mode to work around test suite crashing · fe5da2e6
      Oran Agra authored
      in some cases a command that returns an error possibly due to a timing
      issue causes the tcl code to crash and thus prevents the rest of the
      tests from running. this adds an option to make the test proceed despite
      the crash.
      maybe it should be the default mode some day.
      fe5da2e6
    • Oran Agra's avatar
      test infra - wait_done_loading · 1b7ba44e
      Oran Agra authored
      reduce code duplication in aof.tcl.
      move creation of clients into the test so that it can be skipped
      1b7ba44e
    • Oran Agra's avatar
      b65e5aca
    • Oran Agra's avatar
      test infra - improve test skipping ability · 677d14c2
      Oran Agra authored
      - skip full units
      - skip a single test (not just a list of tests)
      - when skipping tag, skip spinning up servers, not just the tests
      - skip tags when running against an external server too
      - allow using multiple tags (split them)
      677d14c2
    • Oran Agra's avatar
      test infra - reduce disk space usage · e3e69c25
      Oran Agra authored
      this is important when running a test with --loop
      e3e69c25
    • Oran Agra's avatar
      test infra - write test name to logfile · 9d527d07
      Oran Agra authored
      9d527d07
  4. 03 Sep, 2020 3 commits
    • Yossi Gottlieb's avatar
      redis-cli: fix writeConn() buffer handling. (#7749) · 58e5feb3
      Yossi Gottlieb authored
      Fix issues with writeConn() which resulted with corruption of the stream by leaving an extra byte in the buffer. The trigger for this is partial writes or write errors which were not experienced on Linux but reported on macOS.
      58e5feb3
    • WuYunlong's avatar
      fix wrong comments in redis.conf, change default always-show-logo (#5695) · 12f798dc
      WuYunlong authored
      1. default value of always-show-logo was not consistent with the default in the code
      2. comment about cluster-replica-no-failover is wrong since we can only do manually failover upon replicas
      3. improve description about always-show-logo
      12f798dc
    • Oran Agra's avatar
      Run active defrag while blocked / loading (#7726) · 9ef8d2f6
      Oran Agra authored
      During long running scripts or loading RDB/AOF, we may need to do some
      defragging. Since processEventsWhileBlocked is called periodically at
      unknown intervals, and many cron jobs either depend on run_with_period
      (including active defrag), or rely on being called at server.hz rate
      (i.e. active defrag knows ho much time to run by looking at server.hz),
      the whileBlockedCron may have to run a loop triggering the cron jobs in it
      (currently only active defrag) several times.
      
      Other changes:
      - Adding a test for defrag during aof loading.
      - Changing key-load-delay config to take negative values for fractions
        of a microsecond sleep
      9ef8d2f6
  5. 02 Sep, 2020 3 commits
  6. 01 Sep, 2020 2 commits
  7. 31 Aug, 2020 2 commits
  8. 30 Aug, 2020 1 commit
  9. 28 Aug, 2020 2 commits
    • maohuazhu's avatar
      Optimize __ziplistCascadeUpdate algorithm (#6886) · ee4a15aa
      maohuazhu authored
      
      
      The previous algorithm is of O(n^2) time complexity.
      It would have run through the ziplist entries one by one, each time doing a `realloc` and a
      `memmove` (moving the entire tail of the ziplist).
      
      The new algorithm is O(n), it runs over all the records once, computing the size of the `realloc`
      needed, then does one `realloc`, and run thought the records again doing many smaller `memmove`s,
      each time moving just one record.
      
      So this change reduces many reallocs, and moves each record just once.
      Co-authored-by: default avatarzhumaohua <zhumaohua@megvii.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ee4a15aa
    • Jim Brunner's avatar
      Use H/W Monotonic clock and updates to AE (#7644) · c01e94a4
      Jim Brunner authored
      Update adds a general source for retrieving a monotonic time.
      In addition, AE has been updated to utilize the new monotonic
      clock for timer processing.
      
      This performance improvement is **not** enabled in a default build due to various H/W compatibility
      concerns, see README.md for details. It does however change the default use of gettimeofday with
      clock_gettime and somewhat improves performance.
      
      This update provides the following
      1. An interface for retrieving a monotonic clock. getMonotonicUs returns a uint64_t (aka monotime)
         with the number of micro-seconds from an arbitrary point. No more messing with tv_sec/tv_usec.
         Simple routines are provided for measuring elapsed milli-seconds or elapsed micro-seconds (the
         most common use case for a monotonic timer). No worries about time moving backwards.
      2. High-speed assembler implementation for x86 and ARM. The standard method for retrieving the
         monotonic clock is POSIX.1b (1993): cloc...
      c01e94a4
  10. 27 Aug, 2020 4 commits
    • Oran Agra's avatar
      Fix rejectCommand trims newline in shared error objects, hung clients (#7714) · 9fcd9e19
      Oran Agra authored
      65a3307b
      
       (released in 6.0.6) has a side effect, when processCommand
      rejects a command with pre-made shared object error string, it trims the
      newlines from the end of the string. if that string is later used with
      addReply, the newline will be missing, breaking the protocol, and
      leaving the client hung.
      
      It seems that the only scenario which this happens is when replying with
      -LOADING to some command, and later using that reply from the CONFIG
      SET command (still during loading). this will result in hung client.
      
      Refactoring the code in order to avoid trimming these newlines from
      shared string objects, and do the newline trimming only in other cases
      where it's needed.
      Co-authored-by: default avatarGuy Benoish <guy.benoish@redislabs.com>
      9fcd9e19
    • Oran Agra's avatar
      Update memory metrics for INFO during loading (#7690) · 8bdcbbb0
      Oran Agra authored
      During a long AOF or RDB loading, the memory stats were not updated, and
      INFO would return stale data, specifically about fragmentation and RSS.
      In the past some of these were sampled directly inside the INFO command,
      but were moved to cron as an optimization.
      
      This commit introduces a concept of loadingCron which should take
      some of the responsibilities of serverCron.
      It attempts to limit it's rate to approximately the server Hz, but may
      not be very accurate.
      
      In order to avoid too many system call, we use the cached ustime, and
      also make sure to update it in both AOF loading and RDB loading inside
      processEventsWhileBlocked (it seems AOF loading was missing it).
      8bdcbbb0
    • valentinogeron's avatar
      EXEC with only read commands should not be rejected when OOM (#7696) · b7289e91
      valentinogeron authored
      If the server gets MULTI command followed by only read
      commands, and right before it gets the EXEC it reaches OOM,
      the client will get OOM response.
      
      So, from now on, it will get OOM response only if there was
      at least one command that was tagged with `use-memory` flag
      b7289e91
    • Oran Agra's avatar
      Add test coverage for CLIENT UNBLOCK (#7712) · daef1f00
      Oran Agra authored
      plus minor other fixes to list.tcl
      daef1f00
  11. 25 Aug, 2020 2 commits
    • filipe oliveira's avatar
      Extended redis-benchmark instant metrics and overall latency report (#7600) · 21784def
      filipe oliveira authored
      A first step to enable a consistent full percentile analysis on query latency so that we can fully understand the performance and stability characteristics of the redis-server system we are measuring. It also improves the instantaneous reported metrics, and the csv output format.
      21784def
    • Itamar Haber's avatar
      Expands lazyfree's effort estimate to include Streams (#5794) · 5b0a06af
      Itamar Haber authored
      Otherwise, it is treated as a single allocation and freed synchronously. The following logic is used for estimating the effort in constant-ish time complexity:
      
      1. Check the number of nodes.
      1. Add an allocation for each consumer group registered inside the stream.
      1. Check the number of PELs in the first CG, and then add this count times the number of CGs.
      1. Check the number of consumers in the first CG, and then add this count times the number of CGs.
      5b0a06af
  12. 24 Aug, 2020 3 commits
  13. 23 Aug, 2020 3 commits
  14. 21 Aug, 2020 1 commit
  15. 20 Aug, 2020 1 commit