1. 17 Aug, 2020 1 commit
  2. 12 Aug, 2020 2 commits
  3. 11 Aug, 2020 1 commit
    • Tyson Andre's avatar
      Implement SMISMEMBER key member [member ...] (#7615) · 6f11acbd
      Tyson Andre authored
      
      
      This is a rebased version of #3078 originally by shaharmor
      with the following patches by TysonAndre made after rebasing
      to work with the updated C API:
      
      1. Add 2 more unit tests
         (wrong argument count error message, integer over 64 bits)
      2. Use addReplyArrayLen instead of addReplyMultiBulkLen.
      3. Undo changes to src/help.h - for the ZMSCORE PR,
         I heard those should instead be automatically
         generated from the redis-doc repo if it gets updated
      
      Motivations:
      
      - Example use case: Client code to efficiently check if each element of a set
        of 1000 items is a member of a set of 10 million items.
        (Similar to reasons for working on #7593)
      - HMGET and ZMSCORE already exist. This may lead to developers deciding
        to implement functionality that's best suited to a regular set with a
        data type of sorted set or hash map instead, for the multi-get support.
      
      Currently, multi commands or lua scripting to call sismember multiple times
      would almost definitely be less efficient than a native smismember
      for the following reasons:
      
      - Need to fetch the set from the string every time
        instead of reusing the C pointer.
      - Using pipelining or multi-commands would result in more bytes sent
        and received by the client for the repeated SISMEMBER KEY sections.
      - Need to specially encode the data and decode it from the client
        for lua-based solutions.
      - Proposed solutions using Lua or SADD/SDIFF could trigger writes to
        memory, which is undesirable on a redis replica server
        or when commands get replicated to replicas.
      Co-Authored-By: default avatarShahar Mor <shahar@peer5.com>
      Co-Authored-By: default avatarTyson Andre <tysonandre775@hotmail.com>
      6f11acbd
  4. 09 Aug, 2020 2 commits
  5. 06 Aug, 2020 1 commit
    • Oran Agra's avatar
      Accelerate diskless master connections, and general re-connections (#6271) · c17e597d
      Oran Agra authored
      Diskless master has some inherent latencies.
      1) fork starts with delay from cron rather than immediately
      2) replica is put online only after an ACK. but the ACK
         was sent only once a second.
      3) but even if it would arrive immediately, it will not
         register in case cron didn't yet detect that the fork is done.
      
      Besides that, when a replica disconnects, it doesn't immediately
      attempts to re-connect, it waits for replication cron (one per second).
      in case it was already online, it may be important to try to re-connect
      as soon as possible, so that the backlog at the master doesn't vanish.
      
      In case it disconnected during rdb transfer, one can argue that it's
      not very important to re-connect immediately, but this is needed for the
      "diskless loading short read" test to be able to run 100 iterations in 5
      seconds, rather than 3 (waiting for replication cron re-connection)
      
      changes in this commit:
      1) sync command starts a fork immediately if no sync_delay is configured
      2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
      3) when a replica unexpectedly disconnets, it immediately tries to
         re-connect rather than waiting 1s
      4) when when a child exits, if there is another replica waiting, we spawn a new
         one right away, instead of waiting for 1s replicationCron.
      5) added a call to connectWithMaster from replicationSetMaster. which is called
         from the REPLICAOF command but also in 3 places in cluster.c, in all of
         these the connection attempt will now be immediate instead of delayed by 1
         second.
      
      side note:
      we can add a call to rdbPipeReadHandler in replconfCommand when getting
      a REPLCONF ACK from the replica to solve a race where the replica got
      the entire rdb and EOF marker before we detected that the pipe was
      closed.
      in the test i did see this race happens in one about of some 300 runs,
      but i concluded that this race is unlikely in real life (where the
      replica is on another host and we're more likely to first detect the
      pipe was closed.
      the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
      seconds instead (waiting for another REPLCONF ACK).
      
      Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
      Now that CheckChildrenDone is calling the new replicationStartPendingFork
      (extracted from serverCron) there's actually no need to call
      startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
      since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
      calling replicationStartPendingFork that handles that anyway.
      The code in updateSlavesWaitingForBgsave had a bug in which it ignored
      repl-diskless-sync-delay, but removing that code shows that this bug was
      hiding another bug, which is that the max_idle should have used >= and
      not >, this one second delay has a big impact on my new test.
      c17e597d
  6. 04 Aug, 2020 3 commits
    • WuYunlong's avatar
    • Tyson Andre's avatar
      Add a ZMSCORE command returning an array of scores. (#7593) · f11f26cc
      Tyson Andre authored
      
      
      Syntax: `ZMSCORE KEY MEMBER [MEMBER ...]`
      
      This is an extension of #2359
      amended by Tyson Andre to work with the changed unstable API,
      add more tests, and consistently return an array.
      
      - It seemed as if it would be more likely to get reviewed
        after updating the implementation.
      
      Currently, multi commands or lua scripting to call zscore multiple times
      would almost definitely be less efficient than a native ZMSCORE
      for the following reasons:
      
      - Need to fetch the set from the string every time instead of reusing the C
        pointer.
      - Using pipelining or multi-commands would result in more bytes sent by
        the client for the repeated `ZMSCORE KEY` sections.
      - Need to specially encode the data and decode it from the client
        for lua-based solutions.
      - The fastest solution I've seen for large sets(thousands or millions)
        involves lua and a variadic ZADD, then a ZINTERSECT, then a ZRANGE 0 -1,
        then UNLINK of a temporary set (or lua). This is still inefficient.
      Co-authored-by: default avatarTyson Andre <tysonandre775@hotmail.com>
      f11f26cc
    • Oran Agra's avatar
      fix new rdb test failing on timing issues (#7604) · 824bd2ac
      Oran Agra authored
      apparenlty on github actions sometimes 500ms is not enough
      824bd2ac
  7. 31 Jul, 2020 1 commit
  8. 30 Jul, 2020 1 commit
  9. 28 Jul, 2020 3 commits
    • Yossi Gottlieb's avatar
      Fix TLS cluster tests. (#7578) · bedf1b21
      Yossi Gottlieb authored
      Fix consistency test added in af5167b7 without considering TLS
      redis-cli configuration.
      bedf1b21
    • Oran Agra's avatar
      Fix failing tests due to issues with wait_for_log_message (#7572) · 109b5ccd
      Oran Agra authored
      - the test now waits for specific set of log messages rather than wait for
        timeout looking for just one message.
      - we don't wanna sample the current length of the log after an action, due
        to a race, we need to start the search from the line number of the last
        message we where waiting for.
      - when attempting to trigger a full sync, use multi-exec to avoid a race
        where the replica manages to re-connect before we completed the set of
        actions that should force a full sync.
      - fix verify_log_message which was broken and unused
      109b5ccd
    • Jiayuan Chen's avatar
      Add optional tls verification (#7502) · f31260b0
      Jiayuan Chen authored
      
      
      Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      f31260b0
  10. 23 Jul, 2020 2 commits
  11. 21 Jul, 2020 3 commits
    • Oran Agra's avatar
      testsuite may leave servers alive on error (#7549) · 36b94943
      Oran Agra authored
      in cases where you have
      test name {
        start_server {
          start_server {
            assert
          }
        }
      }
      
      the exception will be thrown to the test proc, and the servers are
      supposed to be killed on the way out. but it seems there was always a
      bug of not cleaning the server stack, and recently (#7404) we started
      relying on that stack in order to kill them, so with that bug sometimes
      we would have tried to kill the same server twice, and leave one alive.
      
      luckly, in most cases the pattern is:
      start_server {
        test name {
        }
      }
      36b94943
    • Yossi Gottlieb's avatar
      Tests: drop TCL 8.6 dependency. (#7548) · f57e844b
      Yossi Gottlieb authored
      This re-implements the redis-cli --pipe test so it no longer depends on a close feature available only in TCL 8.6.
      
      Basically what this test does is run redis-cli --pipe, generates a bunch of commands and pipes them through redis-cli, and inspects the result in both Redis and the redis-cli output.
      
      To do that, we need to close stdin for redis-cli to indicate we're done so it can flush its buffers and exit. TCL has bi-directional channels can only offers a way to "one-way close" a channel with TCL 8.6. To work around that, we now generate the commands into a file and feed that file to redis-cli directly.
      
      As we're writing to an actual file, the number of commands is now reduced.
      f57e844b
    • Remi Collet's avatar
      Fix deprecated tail syntax in tests (#7543) · 3f2fbc4c
      Remi Collet authored
      3f2fbc4c
  12. 15 Jul, 2020 1 commit
  13. 14 Jul, 2020 2 commits
  14. 13 Jul, 2020 2 commits
    • Oran Agra's avatar
      fix recently added time sensitive tests failing with valgrind (#7512) · e5227aab
      Oran Agra authored
      interestingly the latency monitor test fails because valgrind is slow
      enough so that the time inside PEXPIREAT command from the moment of
      the first mstime() call to get the basetime until checkAlreadyExpired
      calls mstime() again is more than 1ms, and that test was too sensitive.
      
      using this opportunity to speed up the test (unrelated to the failure)
      the fix is just the longer time passed to PEXPIRE.
      e5227aab
    • Oran Agra's avatar
      runtest --stop pause stops before terminating the redis server (#7513) · 02ef355f
      Oran Agra authored
      in the majority of the cases (on this rarely used feature) we want to
      stop and be able to connect to the shard with redis-cli.
      since these are two different processes interracting with the tty we
      need to stop both, and we'll have to hit enter twice, but it's not that
      bad considering it is rarely used.
      02ef355f
  15. 10 Jul, 2020 7 commits
    • WuYunlong's avatar
    • Yossi Gottlieb's avatar
      TLS: Session caching configuration support. (#7420) · 3e6f2b1a
      Yossi Gottlieb authored
      * TLS: Session caching configuration support.
      * TLS: Remove redundant config initialization.
      3e6f2b1a
    • Yossi Gottlieb's avatar
      TLS: Add missing redis-cli options. (#7456) · d9f970d8
      Yossi Gottlieb authored
      * Tests: fix and reintroduce redis-cli tests.
      
      These tests have been broken and disabled for 10 years now!
      
      * TLS: add remaining redis-cli support.
      
      This adds support for the redis-cli --pipe, --rdb and --replica options
      previously unsupported in --tls mode.
      
      * Fix writeConn().
      d9f970d8
    • Oran Agra's avatar
      RESTORE ABSTTL won't store expired keys into the db (#7472) · 5977a948
      Oran Agra authored
      Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
      key and return success, RESTORE should not store key that are already
      expired into the db.
      When used together with REPLACE it should emit a DEL to keyspace
      notification and replication stream.
      5977a948
    • Oran Agra's avatar
      skip a test that uses +inf on valgrind (#7440) · 909bc97c
      Oran Agra authored
      On some platforms strtold("+inf") with valgrind returns a non-inf result
      
      [err]: INCRBYFLOAT does not allow NaN or Infinity in tests/unit/type/incr.tcl
      Expected 'ERR*would produce*' to equal or match '1189731495357231765085759.....'
      909bc97c
    • Oran Agra's avatar
      stabilize tests that look for log lines (#7367) · 8e76e134
      Oran Agra authored
      tests were sensitive to additional log lines appearing in the log
      causing the search to come empty handed.
      
      instead of just looking for the n last log lines, capture the log lines
      before performing the action, and then search from that offset.
      8e76e134
    • Oran Agra's avatar
      tests/valgrind: don't use debug restart (#7404) · 69ade873
      Oran Agra authored
      * tests/valgrind: don't use debug restart
      
      DEBUG REATART causes two issues:
      1. it uses execve which replaces the original process and valgrind doesn't
         have a chance to check for errors, so leaks go unreported.
      2. valgrind report invalid calls to close() which we're unable to resolve.
      
      So now the tests use restart_server mechanism in the tests, that terminates
      the old server and starts a new one, new PID, but same stdout, stderr.
      
      since the stderr can contain two or more valgrind report, it is not enough
      to just check for the absence of leaks, we also need to check for some known
      errors, we do both, and fail if we either find an error, or can't find a
      report saying there are no leaks.
      
      other changes:
      - when killing a server that was already terminated we check for leaks too.
      - adding DEBUG LEAK which was used to test it.
      - adding --trace-children to valgrind, although no longer needed.
      - since the stdout contains two or more runs, we need slightly different way
        of checking if the new process is up (explicitly looking for the new PID)
      - move the code that handles --wait-server to happen earlier (before
        watching the startup message in the log), and serve the restarted server too.
      
      * squashme - CR fixes
      69ade873
  16. 24 Jun, 2020 1 commit
  17. 23 Jun, 2020 1 commit
    • Oran Agra's avatar
      EXEC always fails with EXECABORT and multi-state is cleared · 65a3307b
      Oran Agra authored
      In order to support the use of multi-exec in pipeline, it is important that
      MULTI and EXEC are never rejected and it is easy for the client to know if the
      connection is still in multi state.
      
      It was easy to make sure MULTI and DISCARD never fail (done by previous
      commits) since these only change the client state and don't do any actual
      change in the server, but EXEC is a different story.
      
      Since in the past, it was possible for clients to handle some EXEC errors and
      retry the EXEC, we now can't affort to return any error on EXEC other than
      EXECABORT, which now carries with it the real reason for the abort too.
      
      Other fixes in this commit:
      - Some checks that where performed at the time of queuing need to be re-
        validated when EXEC runs, for instance if the transaction contains writes
        commands, it needs to be aborted. there was one check that was already done
        in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
        -NOREPLICAS, -MASTERDOWN
      - When a command is rejected by processCommand it was rejected with addReply,
        which was not recognized as an error in case the bad command came from the
        master. this will enable to count or MONITOR these errors in the future.
      - make it easier for tests to create additional (non deferred) clients.
      - add tests for the fixes of this commit.
      65a3307b
  18. 14 Jun, 2020 1 commit
    • meir@redislabs.com's avatar
      Fix RM_ScanKey module api not to return int encoded strings · a89bf734
      meir@redislabs.com authored
      The scan key module API provides the scan callback with the current
      field name and value (if it exists). Those arguments are RedisModuleString*
      which means it supposes to point to robj which is encoded as a string.
      Using createStringObjectFromLongLong function might return robj that
      points to an integer and so break a module that tries for example to
      use RedisModule_StringPtrLen on the given field/value.
      
      The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
      Using those function promise that the field and value pass to the to the
      scan callback will be Strings.
      
      The PR also changes the Scan test module to use RedisModule_StringPtrLen
      to catch the issue. without this, the issue is hidden because
      RedisModule_ReplyWithString knows to handle integer encoding of the
      given robj (RedisModuleString).
      
      The PR also introduces a new test to verify the issue is solved.
      a89bf734
  19. 11 Jun, 2020 1 commit
  20. 09 Jun, 2020 1 commit
  21. 03 Jun, 2020 1 commit
  22. 31 May, 2020 1 commit
  23. 28 May, 2020 1 commit