1. 11 Oct, 2020 1 commit
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 7d117d75
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      7d117d75
  2. 08 Oct, 2020 2 commits
    • Felipe Machado's avatar
      Adds new pop-push commands (LMOVE, BLMOVE) (#6929) · c3f9e017
      Felipe Machado authored
      
      
      Adding [B]LMOVE <src> <dst> RIGHT|LEFT RIGHT|LEFT. deprecating [B]RPOPLPUSH.
      
      Note that when receiving a BRPOPLPUSH we'll still propagate an RPOPLPUSH,
      but on BLMOVE RIGHT LEFT we'll propagate an LMOVE
      
      improvement to existing tests
      - Replace "after 1000" with "wait_for_condition" when wait for
        clients to block/unblock.
      - Add a pre-existing element to target list on basic tests so
        that we can check if the new element was added to the correct
        side of the list.
      - check command stats on the replica to make sure the right
        command was replicated
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      c3f9e017
    • Madelyn Olson's avatar
  3. 06 Oct, 2020 1 commit
    • Oran Agra's avatar
      Allow blocked XREAD on a cluster replica (#7881) · 216c1106
      Oran Agra authored
      I suppose that it was overlooked, since till recently none of the blocked commands were readonly.
      
      other changes:
      - add test for the above.
      - add better support for additional (and deferring) clients for
        cluster tests
      - improve a test which left the client in MULTI state.
      216c1106
  4. 05 Oct, 2020 1 commit
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · bea40e6a
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      bea40e6a
  5. 02 Oct, 2020 1 commit
  6. 01 Oct, 2020 1 commit
  7. 30 Sep, 2020 1 commit
    • nitaicaro's avatar
      Fixed Tracking test “The other connection is able to get invalidations” (#7871) · 8fb89a57
      nitaicaro authored
      
      
      PROBLEM:
      
      [$rd1 read] reads invalidation messages one by one, so it's never going to see the second invalidation message produced after INCR b, whether or not it exists. Adding another read will block incase no invalidation message is produced.
      
      FIX:
      
      We switch the order of "INCR a" and "INCR b" - now "INCR b" comes first. We still only read the first invalidation message produces. If an invalidation message is wrongly produces for b - then it will be produced before that of a, since "INCR b" comes before "INCR a".
      Co-authored-by: default avatarNitai Caro <caronita@amazon.com>
      8fb89a57
  8. 27 Sep, 2020 1 commit
  9. 24 Sep, 2020 3 commits
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 57709c4b
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      57709c4b
    • valentinogeron's avatar
      Stream: Inconsistency between master and replica some XREADGROUP case (#7526) · 795c454d
      valentinogeron authored
      XREADGROUP auto-creates the consumer inside the consumer group the
      first time it saw it.
      When XREADGROUP is being used with NOACK option, the message will not
      be added into the client's PEL and XGROUP SETID would be propagated.
      When the replica gets the XGROUP SETID it will only update the last delivered
      id of the group, but will not create the consumer.
      
      So, in this commit XGROUP CREATECONSUMER is being added.
      Command pattern: XGROUP CREATECONSUMER <key> <group> <consumer>.
      
      When NOACK option is being used, createconsumer command would be
      propagated as well.
      
      In case of AOFREWRITE, consumer with an empty PEL would be saved with
      XGROUP CREATECONSUMER whereas consumer with pending entries would be
      saved with XCLAIM
      795c454d
    • bodong.ybd's avatar
      Add ZINTER/ZUNION command · e08bf166
      bodong.ybd authored
      Syntax: ZINTER/ZUNION numkeys key [key ...] [WEIGHTS weight [weight ...]]
      [AGGREGATE SUM|MIN|MAX] [WITHSCORES]
      
      see #7624
      e08bf166
  10. 23 Sep, 2020 1 commit
  11. 22 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Kill disk-based fork child when all replicas drop and 'save' is not enabled (#7819) · 1bb5794a
      Wang Yuan authored
      When all replicas waiting for a bgsave get disconnected (possibly due to output buffer limit),
      It may be good to kill the bgsave child. in diskless replication it already happens, but in
      disk-based, the child may still serve some purpose (for persistence).
      
      By killing the child, we prevent it from eating COW memory in vain, and we also allow a new child fork sooner for the next full synchronization or bgsave.
      We do that only if rdb persistence wasn't enabled in the configuration.
      
      Btw, now, rdbRemoveTempFile in killRDBChild won't block server, so we can killRDBChild safely.
      1bb5794a
  12. 20 Sep, 2020 1 commit
  13. 17 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Remove tmp rdb file in background thread (#7762) · b002d2b4
      Wang Yuan authored
      We're already using bg_unlink in several places to delete the rdb file in the background,
      and avoid paying the cost of the deletion from our main thread.
      This commit uses bg_unlink to remove the temporary rdb file in the background too.
      
      However, in case we delete that rdb file just before exiting, we don't actually wait for the
      background thread or the main thread to delete it, and just let the OS clean up after us.
      i.e. we open the file, unlink it and exit with the fd still open.
      
      Furthermore, rdbRemoveTempFile can be called from a thread and was using snprintf which is
      not async-signal-safe, we now use ll2string instead.
      b002d2b4
  14. 15 Sep, 2020 1 commit
  15. 13 Sep, 2020 2 commits
    • Mykhailo Pylyp's avatar
    • Oran Agra's avatar
      fix broken PEXPIREAT test (#7791) · ed9bfe22
      Oran Agra authored
      This test was nearly always failing on MacOS github actions.
      This is because of bugs in the test that caused it to nearly always run
      all 3 attempts and just look at the last one as the pass/fail creteria.
      
      i.e. the test was nearly always running all 3 attempts and still sometimes
      succeed. this is because the break condition was different than the test
      completion condition.
      
      The reason the test succeeded is because the break condition tested the
      results of all 3 tests (PSETEX/PEXPIRE/PEXPIREAT), but the success check
      at the end was only testing the result of PSETEX.
      
      The reason the PEXPIREAT test nearly always failed is because it was
      getting the current time wrong: getting the current second and loosing
      the sub-section time, so the only chance for it to succeed is if it run
      right when a certain second started.
      
      Because i now get the time from redis, adding another round trip, i
      added another 100ms to the PEXPIRE test to make it less fragile, and
      also added many more attempts.
      
      Adding many more attempts before failure to account for slow platforms,
      github actions and valgrind
      ed9bfe22
  16. 10 Sep, 2020 2 commits
    • Oran Agra's avatar
      Squash merging 125 typo/grammar/comment/doc PRs (#7773) · 1c710385
      Oran Agra authored
      List of squashed commits or PRs
      ===============================
      
      commit 66801ea
      Author: hwware <wen.hui.ware@gmail.com>
      Date:   Mon Jan 13 00:54:31 2020 -0500
      
          typo fix in acl.c
      
      commit 46f55db
      Author: Itamar Haber <itamar@redislabs.com>
      Date:   Sun Sep 6 18:24:11 2020 +0300
      
          Updates a couple of comments
      
          Specifically:
      
          * RM_AutoMemory completed instead of pointing to docs
          * Updated link to custom type doc
      
      commit 61a2aa0
      Author: xindoo <xindoo@qq.com>
      Date:   Tue Sep 1 19:24:59 2020 +0800
      
          Correct errors in code comments
      
      commit a5871d1
      Author: yz1509 <pro-756@qq.com>
      Date:   Tue Sep 1 18:36:06 2020 +0800
      
          fix typos in module.c
      
      commit 41eede7
      Author: bookug <bookug@qq.com>
      Date:   Sat Aug 15 01:11:33 2020 +0800
      
          docs: fix typos in comments
      
      commit c303c84
      Author: lazy-snail <ws.niu@outlook.com>
      Date:   Fri Aug 7 11:15:44 2020 +0800
      
          fix spelling in redis.conf
      
      commit 1e...
      1c710385
    • Oran Agra's avatar
      8ea131fc
  17. 09 Sep, 2020 7 commits
  18. 08 Sep, 2020 2 commits
    • Oran Agra's avatar
      handle cur_test for nested tests · 0a1e7341
      Oran Agra authored
      if there are nested tests and nested servers, we need to restore the
      previous value of cur_test when a test exist.
      
      example:
      ```
      test{test 1} {
      	start_server {
      		test{test 1.1 - master only} {
      		}
      		start_server {
      		    test{test 1.2 - with replication} {
                  }
      		}
      	}
      }
      ```
      when `test 1.1 - master only exists`, we're still inside `test 1`
      0a1e7341
    • bodong.ybd's avatar
      Tests: Some fixes for macOS · f22fa959
      bodong.ybd authored
      1) cur_test: when restart_server, "no such variable" error occurs
        ./runtest --single integration/rdb
        test {client freed during loading}
            SET ::cur_test
            restart_server
              kill_server
                test "Check for memory leaks (pid $pid)"
                SET ::cur_test
                UNSET ::cur_test
            UNSET ::cur_test // This global variable has been unset.
      
      2) `ps --ppid` not available on macOS platform, can be replaced with
      `pgrep -P pid`.
      f22fa959
  19. 07 Sep, 2020 3 commits
    • Oran Agra's avatar
      Fix cluster consistency-check test (#7754) · b491d477
      Oran Agra authored
      This test was failing from time to time see discussion at the bottom of #7635
      This was probably due to timing, the DEBUG SLEEP executed by redis-cli
      didn't sleep for enough time.
      
      This commit changes:
      1) use SET-ACTIVE-EXPIRE instead of DEBUG SLEEP
      2) reduce many `after` sleeps with retry loops to speed up the test.
      3) add many comment explaining the different steps of the test and
         it's purpose.
      4) config appendonly before populating the volatile keys, so that they'll
         be part of the AOF command stream rather than the preamble RDB portion.
      
      other complications: recently kill_instance switched from SIGKILL to
      SIGTERM, and this would sometimes fail since there was an AOFRW running
      in the background. now we wait for it to end before attempting the kill.
      b491d477
    • Yossi Gottlieb's avatar
      Tests: fix unmonitored servers. (#7756) · 2df4cb93
      Yossi Gottlieb authored
      There is an inherent race condition in port allocation for spawned
      servers. If a server fails to start because a port is taken, a new port
      is allocated. This fixes a problem where the logs are not truncated and
      as a result a large number of unmonitored servers are started.
      2df4cb93
    • Oran Agra's avatar
      fix broken cluster/sentinel tests by recent commit (#7752) · 42ba7a1b
      Oran Agra authored
      2b998de4 added a file for stderr to keep valgrind log but i forgot to
      add a similar thing when valgrind isn't being used.
      the result is that `glob */err.txt` fails.
      42ba7a1b
  20. 06 Sep, 2020 7 commits
    • Oran Agra's avatar
      if diskless repl child is killed, make sure to reap the pid (#7742) · 573246f7
      Oran Agra authored
      Starting redis 6.0 and the changes we made to the diskless master to be
      suitable for TLS, I made the master avoid reaping (wait3) the pid of the
      child until we know all replicas are done reading their rdb.
      
      I did that in order to avoid a state where the rdb_child_pid is -1 but
      we don't yet want to start another fork (still busy serving that data to
      replicas).
      
      It turns out that the solution used so far was problematic in case the
      fork child was being killed (e.g. by the kernel OOM killer), in that
      case there's a chance that we currently disabled the read event on the
      rdb pipe, since we're waiting for a replica to become writable again.
      and in that scenario the master would have never realized the child
      exited, and the replica will remain hung too.
      Note that there's no mechanism to detect a hung replica while it's in
      rdb transfer state.
      
      The solution here is to add another pipe which is used by the parent to
      tell the child it is safe to exit. this mean that when the child exits,
      for whatever reason, it is safe to reap it.
      
      Besides that, i'm re-introducing an adjustment to REPLCONF ACK which was
      part of #6271 (Accelerate diskless master connections) but was dropped
      when that PR was rebased after the TLS fork/pipe changes (5a477946).
      Now that RdbPipeCleanup no longer calls checkChildrenDone, and the ACK
      has chance to detect that the child exited, it should be the one to call
      it so that we don't have to wait for cron (server.hz) to do that.
      573246f7
    • Oran Agra's avatar
      Improve valgrind support for cluster tests (#7725) · 2b998de4
      Oran Agra authored
      - redirect valgrind reports to a dedicated file rather than console
      - try to avoid killing instances with SIGKILL so that we get the memory
        leak report (killing with SIGTERM before resorting to SIGKILL)
      - search for valgrind reports when done, print them and fail the tests
      - add --dont-clean option to keep the logs on exit
      - fix exit error code when crash is found (would have exited with 0)
      
      changes that affect the normal redis test suite:
      - refactor check_valgrind_errors into two functions one to search and
        one to report
      - move the search half into util.tcl to serve the cluster tests too
      - ignore "address range perms" valgrind warnings which seem non relevant.
      2b998de4
    • Oran Agra's avatar
      test infra - add durable mode to work around test suite crashing · fe5da2e6
      Oran Agra authored
      in some cases a command that returns an error possibly due to a timing
      issue causes the tcl code to crash and thus prevents the rest of the
      tests from running. this adds an option to make the test proceed despite
      the crash.
      maybe it should be the default mode some day.
      fe5da2e6
    • Oran Agra's avatar
      test infra - wait_done_loading · 1b7ba44e
      Oran Agra authored
      reduce code duplication in aof.tcl.
      move creation of clients into the test so that it can be skipped
      1b7ba44e
    • Oran Agra's avatar
      b65e5aca
    • Oran Agra's avatar
      test infra - improve test skipping ability · 677d14c2
      Oran Agra authored
      - skip full units
      - skip a single test (not just a list of tests)
      - when skipping tag, skip spinning up servers, not just the tests
      - skip tags when running against an external server too
      - allow using multiple tags (split them)
      677d14c2
    • Oran Agra's avatar
      test infra - reduce disk space usage · e3e69c25
      Oran Agra authored
      this is important when running a test with --loop
      e3e69c25