1. 22 Jun, 2021 1 commit
    • Oran Agra's avatar
      solve test timing issues in replication tests (#9121) · d0819d61
      Oran Agra authored
      # replication-3.tcl
      had a test timeout failure with valgrind on daily CI:
      ```
      *** [err]: SLAVE can reload "lua" AUX RDB fields of duplicated scripts in tests/integration/replication-3.tcl
      Replication not started.
      ```
      replication took more than 70 seconds.
      https://github.com/redis/redis/runs/2854037905?check_suite_focus=true
      
      on my machine it takes only about 30, but i can see how 50 seconds isn't enough.
      
      # replication.tcl
      loading was over too quickly in freebsd daily CI:
      ```
      *** [err]: slave fails full sync and diskless load swapdb recovers it in tests/integration/replication.tcl
      Expected '0' to be equal to '1' (context: type eval line 44 cmd {assert_equal [s -1 loading] 1} proc ::start_server)
      ```
      
      # rdb.tcl
      loading was over too quickly.
      increase the time loading takes, and decrease the amount of work we try to achieve in that time.
      d0819d61
  2. 14 Jun, 2021 1 commit
    • YaacovHazan's avatar
      cleanup around loadAppendOnlyFile (#9012) · 1677efb9
      YaacovHazan authored
      Today when we load the AOF on startup, the loadAppendOnlyFile checks if
      the file is openning for reading.
      This check is redundent (dead code) as we open the AOF file for writing at initServer,
      and the file will always be existing for the loadAppendOnlyFile.
      
      In this commit:
      - remove all the exit(1) from loadAppendOnlyFile, as it is the caller
        responsibility to decide what to do in case of failure.
      - move the opening of the AOF file for writing, to be after we loading it.
      - avoid return -ERR in DEBUG LOADAOF, when the AOF is existing but empty
      1677efb9
  3. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  4. 09 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve test suite to handle external servers better. (#9033) · 8a86bca5
      Yossi Gottlieb authored
      This commit revives the improves the ability to run the test suite against
      external servers, instead of launching and managing `redis-server` processes as
      part of the test fixture.
      
      This capability existed in the past, using the `--host` and `--port` options.
      However, it was quite limited and mostly useful when running a specific tests.
      Attempting to run larger chunks of the test suite experienced many issues:
      
      * Many tests depend on being able to start and control `redis-server` themselves,
      and there's no clear distinction between external server compatible and other
      tests.
      * Cluster mode is not supported (resulting with `CROSSSLOT` errors).
      
      This PR cleans up many things and makes it possible to run the entire test suite
      against an external server. It also provides more fine grained controls to
      handle cases where the external server supports a subset of the Redis commands,
      limited number of databases, cluster mode, etc.
      
      The tests directory now contains a `README.md` file that describes how this
      works.
      
      This commit also includes additional cleanups and fixes:
      
      * Tests can now be tagged.
      * Tag-based selection is now unified across `start_server`, `tags` and `test`.
      * More information is provided about skipped or ignored tests.
      * Repeated patterns in tests have been extracted to common procedures, both at a
        global level and on a per-test file basis.
      * Cleaned up some cases where test setup was based on a previous test executing
        (a major anti-pattern that repeats itself in many places).
      * Cleaned up some cases where test teardown was not part of a test (in the
        future we should have dedicated teardown code that executes even when tests
        fail).
      * Fixed some tests that were flaky running on external servers.
      8a86bca5
  5. 03 Jun, 2021 1 commit
  6. 30 May, 2021 1 commit
    • ny0312's avatar
      Always replicate time-to-live(TTL) as absolute timestamps in milliseconds (#8474) · 53d1acd5
      ny0312 authored
      Till now, on replica full-sync we used to transfer absolute time for TTL,
      however when a command arrived (EXPIRE or EXPIREAT),
      we used to propagate it as is to replicas (possibly with relative time),
      but always translate it to EXPIREAT (absolute time) to AOF.
      
      This commit changes that and will always use absolute time for propagation.
      see discussion in #8433
      
      Furthermore, we Introduce new commands: `EXPIRETIME/PEXPIRETIME`
      that allow extracting the absolute TTL time from a key.
      53d1acd5
  7. 26 May, 2021 1 commit
    • YaacovHazan's avatar
      unregister AE_READABLE from the read pipe in backgroundSaveDoneHandlerSocket (#8991) · 501d7755
      YaacovHazan authored
      In diskless replication, we create a read pipe for the RDB, between the child and the parent.
      When we close this pipe (fd), the read handler also needs to be removed from the event loop (if it still registered).
      Otherwise, next time we will use the same fd, the registration will be fail (panic), because
      we will use EPOLL_CTL_MOD (the fd still register in the event loop), on fd that already removed from epoll_ctl
      501d7755
  8. 20 May, 2021 1 commit
    • YaacovHazan's avatar
      stabilize tests that involved with load handlers (#8967) · 32a2584e
      YaacovHazan authored
      When test stop 'load handler' by killing the process that generating the load,
      some commands that already in the input buffer, still might be processed by the server.
      This may cause some instability in tests, that count on that no more commands
      processed after we stop the `load handler'
      
      In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
      cammands from any 'load handler' prossesed, by checking that the clients who
      genreate the load is disconnceted.
      
      Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
      it would fail in case the last key the workload wrote was an overridden key (not a new one).
      
      Affected tests
      Race fix:
      - failover command to specific replica works
      - Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
      - AOF rewrite during write load: RDB preamble=$rdbpre
      
      Cleanup and speedup:
      - Test replication with blocking lists and sorted sets operations
      - Test replication with parallel clients writing in different DBs
      - Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect
      32a2584e
  9. 18 May, 2021 1 commit
  10. 25 Apr, 2021 1 commit
  11. 20 Apr, 2021 1 commit
  12. 18 Apr, 2021 1 commit
    • Oran Agra's avatar
      Fix timing of new replication test (#8807) · a9897b00
      Oran Agra authored
      In github actions CI with valgrind, i saw that even the fast replica
      (one that wasn't paused), didn't get to complete the replication fast
      enough, and ended up getting disconnected by timeout.
      
      Additionally, due to a typo in uname, we didn't get to actually run the
      CPU efficiency part of the test.
      a9897b00
  13. 15 Apr, 2021 1 commit
    • guybe7's avatar
      Add a timeout mechanism for replicas stuck in fullsync (#8762) · d63d0260
      guybe7 authored
      Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork
      child so that the parent is the one sending data to the replicas.
      This mechanism has an issue in which a hung replica will cause the master to wait
      for it to read the data sent to it forever, thus preventing the fork child from terminating
      and preventing the creations of any other forks.
      
      This PR adds a timeout mechanism, much like the ACK-based timeout,
      we disconnect replicas that aren't reading the RDB file fast enough.
      d63d0260
  14. 30 Mar, 2021 1 commit
    • Oran Agra's avatar
      solve race conditions in psync2-pingoff test (#8720) · cd81dcf1
      Oran Agra authored
      Another test race condition in the macos tests.
      the test was waiting for PINGs to be generated and put on the replication stream,
      but waiting for 1 or 2 seconds doesn't really guarantee that.
      then the test that expected 6 full syncs, found only 4
      cd81dcf1
  15. 24 Mar, 2021 2 commits
    • Qu Chen's avatar
      Properly initialize variable to make valgrind happy in checkChildrenDone().... · 7de64518
      Qu Chen authored
      Properly initialize variable to make valgrind happy in checkChildrenDone(). Removed usage for the obsolete wait3() and wait4() in favor of waitpid(), and properly check for the exit status code. (#8666)
      
      7de64518
    • Oran Agra's avatar
      Corrupt stream key access to uninitialized memory (#8681) · f6e1a94e
      Oran Agra authored
      the corrupt-dump-fuzzer test found a case where an access to a corrupt
      stream would have caused accessing to uninitialized memory.
      now it'll panic instead.
      
      The issue was that there was a stream that says it has more than 0
      records, but looking for the max ID came back empty handed.
      
      p.s. when sanitize-dump-payload is used, this corruption is detected,
      and the RESTORE command is gracefully rejected.
      f6e1a94e
  16. 22 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix race in replication test (#8679) · a7c02b19
      Oran Agra authored
      Since redis 6.2, redis immediately tries to connect to the master, not
      waiting for replication cron.
      
      in the slow freebsd CI, this test failed and master_link_status was
      already "up" when INFO was called.
      a7c02b19
  17. 04 Mar, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve redis-cli non-binary safe string handling. (#8566) · 3c7d6a18
      Yossi Gottlieb authored
      * The `redis-cli --scan` output should honor output mode (set explicitly or implicitly), and quote key names when not in raw mode.
        * Technically this is a breaking change, but it should be very minor since raw mode is by default on for non-tty output.
        * It should only affect  TTY output (human users) or non-tty output if `--no-raw` is specified.
      
      * Added `--quoted-input` option to treat all arguments as potentially quoted strings.
      * Added `--quoted-pattern` option to accept a potentially quoted pattern.
      
      Unquoting is applied to potentially quoted input only if single or double quotes are used. 
      
      Fixes #8561, #8563
      3c7d6a18
  18. 02 Mar, 2021 1 commit
  19. 01 Mar, 2021 1 commit
  20. 23 Feb, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix failed tests on Linux Alpine and add a CI job. (#8532) · 95ea7454
      Yossi Gottlieb authored
      * Remove linux/version.h dependency.
      
      This introduces unnecessary dependencies, and generally not a good idea
      as the platform we build on may be different than the platform we run
      on.
      
      To determine if sync_file_range exists we can simply rely on header file
      hints.
      
      * Fix setproctitle() on libmusl.
      
      The previous ifdef checks were a bit too strict for no apparent
      reason.
      
      * Fix tests failure on Linux with no backtrace.
      
      * Add alpine daily CI job.
      95ea7454
  21. 16 Feb, 2021 1 commit
    • uriyage's avatar
      Adds INFO fields to track fork child progress (#8414) · fd052d2a
      uriyage authored
      * Adding current_save_keys_total and current_save_keys_processed info fields.
        Present in replication, BGSAVE and AOFRW.
      * Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
      * Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
        and module forks.
      fd052d2a
  22. 15 Feb, 2021 2 commits
  23. 07 Feb, 2021 1 commit
  24. 03 Feb, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix FreeBSD tests and CI Daily issues. (#8438) · de6f3ad0
      Yossi Gottlieb authored
      * Add bash temporarily to allow sentinel fd leaks test to run.
      * Use vmactions-freebsd rdist sync to work around bind permission denied
        and slow execution issues.
      * Upgrade to tcl8.6 to be aligned with latest Ubuntu envs.
      * Concat all command executions to avoid ignoring failures.
      * Skip intensive fuzzer on FreeBSD. For some yet unknown reason, generate_fuzzy_traffic_on_key causes TCL to significantly bloat on FreeBSD resulting with out of memory.
      de6f3ad0
  25. 31 Jan, 2021 1 commit
    • Oran Agra's avatar
      Fix test issues from introduction of HRANDFIELD (#8424) · 5a7eb9c8
      Oran Agra authored
      * The corrupt dump fuzzer found a division by zero.
      * in some cases the random fields from the HRANDFIELD tests produced
        fields with newlines and other special chars (due to \ char), this caused
        the TCL tests to see a bulk response that has a newline in it and add {}
        around it, later it can think this is a nested list. in fact the `alpha` random
        string generator isn't using spaces and newlines, so it should not use `\`
        either.
      5a7eb9c8
  26. 28 Jan, 2021 1 commit
  27. 27 Jan, 2021 1 commit
    • Raghav Muddur's avatar
      GETEX, GETDEL and SET PXAT/EXAT (#8327) · 0367a808
      Raghav Muddur authored
      This commit introduces two new command and two options for an existing command
      
      GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
      [PXAT milliseconds-timestamp]
      
      The getexCommand() function implements extended options and variants of the GET
      command. Unlike GET command this command is not read-only. Only one of the options
      can be used at a given time.
      
      1. PERSIST removes any TTL associated with the key.
      2. EX Set expiry TTL in seconds.
      3. PX Set expiry TTL in milliseconds.
      4. EXAT Same like EX instead of specifying the number of seconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      5. PXAT Same like PX instead of specifying the number of milliseconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      
      Command would return either the bulk string, error or nil.
      
      GETDEL <key>
      Would delete the key after getting.
      
      SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] ...
      0367a808
  28. 17 Jan, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add io-thread daily CI tests. (#8232) · 522d9360
      Yossi Gottlieb authored
      This adds basic coverage to IO threads by running the cluster and few selected Redis test suite tests with the IO threads enabled.
      
      Also provides some necessary additional improvements to the test suite:
      
      * Add --config to sentinel/cluster tests for arbitrary configuration.
      * Fix --tags whitelisting which was broken.
      * Add a `network` tag to some tests that are more network intensive. This is work in progress and more tests should be properly tagged in the future.
      522d9360
  29. 08 Jan, 2021 1 commit
    • Oran Agra's avatar
      Fix last COW INFO report, Skip test on non-linux platforms (#8301) · 8dd16cae
      Oran Agra authored
      - the last COW report wasn't always read from the pipe
        (receiveLastChildInfo wasn't used)
      - but in fact, there's no reason we won't always try to drain that pipe
        so i'm unifying receiveLastChildInfo with receiveChildInfo
      - adjust threshold of the COW test when run in accurate mode
      - add some prints in case this test fails again
      - fix indentation, page size, and PID! in MacOS proc info
      
      p.s. it seems that pri_pages_dirtied is always 0
      8dd16cae
  30. 07 Jan, 2021 1 commit
    • YaacovHazan's avatar
      Report child copy-on-write info continuously · ea930a35
      YaacovHazan authored
      Add INFO field, rdb_active_cow_size, to report COW of a live fork child while
      it's active.
      - once in 1024 keys check the time, and if there's more than one second since
        the last report send a report to the parent via the pipe.
      - refactor the child_info_data struct, it's an implementation detail that
        shouldn't be in the server struct, and not used to communicate data between
        caller and callee
      - remove the magic value from that struct (not sure what it was good for), and
        instead add handling of short reads.
      - add another value to the structure, cow_type, to indicate if the report is
        for the new rdb_active_cow_size field, or it's the last report of a
        successful operation
      - add new Module API to report the active COW
      - add more asserts variants to test.tcl
      ea930a35
  31. 14 Dec, 2020 1 commit
  32. 06 Dec, 2020 5 commits
    • Oran Agra's avatar
      Sanitize dump payload: fail RESTORE if memory allocation fails · 7ca00d69
      Oran Agra authored
      When RDB input attempts to make a huge memory allocation that fails,
      RESTORE should fail gracefully rather than die with panic
      7ca00d69
    • Oran Agra's avatar
      Sanitize dump payload: validate no duplicate records in hash/zset/intset · 3716950c
      Oran Agra authored
      If RESTORE passes successfully with full sanitization, we can't affort
      to crash later on assertion due to duplicate records in a hash when
      converting it form ziplist to dict.
      This means that when doing full sanitization, we must make sure there
      are no duplicate records in any of the collections.
      3716950c
    • Oran Agra's avatar
      Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed · c31055db
      Oran Agra authored
      The test creates keys with various encodings, DUMP them, corrupt the payload
      and RESTORES it.
      It utilizes the recently added use-exit-on-panic config to distinguish between
       asserts and segfaults.
      If the restore succeeds, it runs random commands on the key to attempt to
      trigger a crash.
      
      It runs in two modes, one with deep sanitation enabled and one without.
      In the first one we don't expect any assertions or segfaults, in the second one
      we expect assertions, but no segfaults.
      We also check for leaks and invalid reads using valgrind, and if we find them
      we print the commands that lead to that issue.
      
      Changes in the code (other than the test):
      - Replace a few NPD (null pointer deference) flows and division by zero with an
        assertion, so that it doesn't fail the test. (since we set the server to use
        `exit` rather than `abort` on assertion).
      - Fix quite a lot of flows in rdb.c that could have lead to memory leaks in
        RESTORE command (since it now responds with an error rather than panic)
      - Add a DEBUG flag for SET-SKIP-CHECKSUM-VALIDATION so that the test don't need
        to bother with faking a valid checksum
      - Remove a pile of code in serverLogObjectDebugInfo which is actually unsafe to
        run in the crash report (see comments in the code)
      - fix a missing boundary check in lzf_decompress
      
      test suite infra improvements:
      - be able to run valgrind checks before the process terminates
      - rotate log files when restarting servers
      c31055db
    • Oran Agra's avatar
      Sanitize dump payload: improve tests of ziplist and stream encodings · 01c13bdd
      Oran Agra authored
      - improve stream rdb encoding test to include more types of stream metadata
      - add test to cover various ziplist encoding entries (although it does
        look like the stress test above it is able to find some too
      - add another test for ziplist encoding for hash with full sanitization
      - add similar ziplist encoding tests for list
      01c13bdd
    • Oran Agra's avatar
      Sanitize dump payload: ziplist, listpack, zipmap, intset, stream · ca1c1825
      Oran Agra authored
      When loading an encoded payload we will at least do a shallow validation to
      check that the size that's encoded in the payload matches the size of the
      allocation.
      This let's us later use this encoded size to make sure the various offsets
      inside encoded payload don't reach outside the allocation, if they do, we'll
      assert/panic, but at least we won't segfault or smear memory.
      
      We can also do 'deep' validation which runs on all the records of the encoded
      payload and validates that they don't contain invalid offsets. This lets us
      detect corruptions early and reject a RESTORE command rather than accepting
      it and asserting (crashing) later when accessing that payload via some command.
      
      configuration:
      - adding ACL flag skip-sanitize-payload
      - adding config sanitize-dump-payload [yes/no/clients]
      
      For now, we don't have a good way to ensure MIGRATE in cluster resharding isn't
      being slowed down by these sanitation, so i'm setting the default value to `...
      ca1c1825
  33. 02 Dec, 2020 1 commit
    • luhuachao's avatar
      Modify help msg PING_BULK to PING_MBULK in benchmark (#8109) · 7885faf1
      luhuachao authored
      As described in redis-benchamrk help message 'The test names are the same as the ones produced as output.', In redis-benchmark output, we can only see PING_BULK, but the cmd `redis-benchmark -t ping_bulk` is not supported. We  have to run it with ping_mbulk which is not user friendly.
      7885faf1
  34. 04 Nov, 2020 1 commit