1. 06 Aug, 2020 1 commit
    • Oran Agra's avatar
      Accelerate diskless master connections, and general re-connections (#6271) · c17e597d
      Oran Agra authored
      Diskless master has some inherent latencies.
      1) fork starts with delay from cron rather than immediately
      2) replica is put online only after an ACK. but the ACK
         was sent only once a second.
      3) but even if it would arrive immediately, it will not
         register in case cron didn't yet detect that the fork is done.
      
      Besides that, when a replica disconnects, it doesn't immediately
      attempts to re-connect, it waits for replication cron (one per second).
      in case it was already online, it may be important to try to re-connect
      as soon as possible, so that the backlog at the master doesn't vanish.
      
      In case it disconnected during rdb transfer, one can argue that it's
      not very important to re-connect immediately, but this is needed for the
      "diskless loading short read" test to be able to run 100 iterations in 5
      seconds, rather than 3 (waiting for replication cron re-connection)
      
      changes in this commit:
      1) sync command starts a fork immediately if no sync_delay is configured
      2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
      3) when a replica unexpectedly disconnets, it immediately tries to
         re-connect rather than waiting 1s
      4) when when a child exits, if there is another replica waiting, we spawn a new
         one right away, instead of waiting for 1s replicationCron.
      5) added a call to connectWithMaster from replicationSetMaster. which is called
         from the REPLICAOF command but also in 3 places in cluster.c, in all of
         these the connection attempt will now be immediate instead of delayed by 1
         second.
      
      side note:
      we can add a call to rdbPipeReadHandler in replconfCommand when getting
      a REPLCONF ACK from the replica to solve a race where the replica got
      the entire rdb and EOF marker before we detected that the pipe was
      closed.
      in the test i did see this race happens in one about of some 300 runs,
      but i concluded that this race is unlikely in real life (where the
      replica is on another host and we're more likely to first detect the
      pipe was closed.
      the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
      seconds instead (waiting for another REPLCONF ACK).
      
      Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
      Now that CheckChildrenDone is calling the new replicationStartPendingFork
      (extracted from serverCron) there's actually no need to call
      startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
      since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
      calling replicationStartPendingFork that handles that anyway.
      The code in updateSlavesWaitingForBgsave had a bug in which it ignored
      repl-diskless-sync-delay, but removing that code shows that this bug was
      hiding another bug, which is that the max_idle should have used >= and
      not >, this one second delay has a big impact on my new test.
      c17e597d
  2. 04 Aug, 2020 1 commit
    • Tyson Andre's avatar
      Add a ZMSCORE command returning an array of scores. (#7593) · f11f26cc
      Tyson Andre authored
      
      
      Syntax: `ZMSCORE KEY MEMBER [MEMBER ...]`
      
      This is an extension of #2359
      amended by Tyson Andre to work with the changed unstable API,
      add more tests, and consistently return an array.
      
      - It seemed as if it would be more likely to get reviewed
        after updating the implementation.
      
      Currently, multi commands or lua scripting to call zscore multiple times
      would almost definitely be less efficient than a native ZMSCORE
      for the following reasons:
      
      - Need to fetch the set from the string every time instead of reusing the C
        pointer.
      - Using pipelining or multi-commands would result in more bytes sent by
        the client for the repeated `ZMSCORE KEY` sections.
      - Need to specially encode the data and decode it from the client
        for lua-based solutions.
      - The fastest solution I've seen for large sets(thousands or millions)
        involves lua and a variadic ZADD, then a ZINTERSECT, then a ZRANGE 0 -1,
        then UNLINK of a temporary set (or lua). This is still inefficient.
      Co-authored-by: default avatarTyson Andre <tysonandre775@hotmail.com>
      f11f26cc
  3. 31 Jul, 2020 1 commit
  4. 28 Jul, 2020 2 commits
    • Oran Agra's avatar
      Fix failing tests due to issues with wait_for_log_message (#7572) · 109b5ccd
      Oran Agra authored
      - the test now waits for specific set of log messages rather than wait for
        timeout looking for just one message.
      - we don't wanna sample the current length of the log after an action, due
        to a race, we need to start the search from the line number of the last
        message we where waiting for.
      - when attempting to trigger a full sync, use multi-exec to avoid a race
        where the replica manages to re-connect before we completed the set of
        actions that should force a full sync.
      - fix verify_log_message which was broken and unused
      109b5ccd
    • Jiayuan Chen's avatar
      Add optional tls verification (#7502) · f31260b0
      Jiayuan Chen authored
      
      
      Adds an `optional` value to the previously boolean `tls-auth-clients` configuration keyword.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      f31260b0
  5. 23 Jul, 2020 1 commit
  6. 15 Jul, 2020 1 commit
  7. 13 Jul, 2020 1 commit
    • Oran Agra's avatar
      fix recently added time sensitive tests failing with valgrind (#7512) · e5227aab
      Oran Agra authored
      interestingly the latency monitor test fails because valgrind is slow
      enough so that the time inside PEXPIREAT command from the moment of
      the first mstime() call to get the basetime until checkAlreadyExpired
      calls mstime() again is more than 1ms, and that test was too sensitive.
      
      using this opportunity to speed up the test (unrelated to the failure)
      the fix is just the longer time passed to PEXPIRE.
      e5227aab
  8. 10 Jul, 2020 4 commits
    • Yossi Gottlieb's avatar
      TLS: Session caching configuration support. (#7420) · 3e6f2b1a
      Yossi Gottlieb authored
      * TLS: Session caching configuration support.
      * TLS: Remove redundant config initialization.
      3e6f2b1a
    • Oran Agra's avatar
      RESTORE ABSTTL won't store expired keys into the db (#7472) · 5977a948
      Oran Agra authored
      Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
      key and return success, RESTORE should not store key that are already
      expired into the db.
      When used together with REPLACE it should emit a DEL to keyspace
      notification and replication stream.
      5977a948
    • Oran Agra's avatar
      skip a test that uses +inf on valgrind (#7440) · 909bc97c
      Oran Agra authored
      On some platforms strtold("+inf") with valgrind returns a non-inf result
      
      [err]: INCRBYFLOAT does not allow NaN or Infinity in tests/unit/type/incr.tcl
      Expected 'ERR*would produce*' to equal or match '1189731495357231765085759.....'
      909bc97c
    • Oran Agra's avatar
      stabilize tests that look for log lines (#7367) · 8e76e134
      Oran Agra authored
      tests were sensitive to additional log lines appearing in the log
      causing the search to come empty handed.
      
      instead of just looking for the n last log lines, capture the log lines
      before performing the action, and then search from that offset.
      8e76e134
  9. 24 Jun, 2020 1 commit
  10. 23 Jun, 2020 1 commit
    • Oran Agra's avatar
      EXEC always fails with EXECABORT and multi-state is cleared · 65a3307b
      Oran Agra authored
      In order to support the use of multi-exec in pipeline, it is important that
      MULTI and EXEC are never rejected and it is easy for the client to know if the
      connection is still in multi state.
      
      It was easy to make sure MULTI and DISCARD never fail (done by previous
      commits) since these only change the client state and don't do any actual
      change in the server, but EXEC is a different story.
      
      Since in the past, it was possible for clients to handle some EXEC errors and
      retry the EXEC, we now can't affort to return any error on EXEC other than
      EXECABORT, which now carries with it the real reason for the abort too.
      
      Other fixes in this commit:
      - Some checks that where performed at the time of queuing need to be re-
        validated when EXEC runs, for instance if the transaction contains writes
        commands, it needs to be aborted. there was one check that was already done
        in execCommand (-READONLY), but other checks where missing: -OOM, -MISCONF,
        -NOREPLICAS, -MASTERDOWN
      - When a command is rejected by processCommand it was rejected with addReply,
        which was not recognized as an error in case the bad command came from the
        master. this will enable to count or MONITOR these errors in the future.
      - make it easier for tests to create additional (non deferred) clients.
      - add tests for the fixes of this commit.
      65a3307b
  11. 14 Jun, 2020 1 commit
    • meir@redislabs.com's avatar
      Fix RM_ScanKey module api not to return int encoded strings · a89bf734
      meir@redislabs.com authored
      The scan key module API provides the scan callback with the current
      field name and value (if it exists). Those arguments are RedisModuleString*
      which means it supposes to point to robj which is encoded as a string.
      Using createStringObjectFromLongLong function might return robj that
      points to an integer and so break a module that tries for example to
      use RedisModule_StringPtrLen on the given field/value.
      
      The PR introduces a fix that uses the createObject function and sdsfromlonglong function.
      Using those function promise that the field and value pass to the to the
      scan callback will be Strings.
      
      The PR also changes the Scan test module to use RedisModule_StringPtrLen
      to catch the issue. without this, the issue is hidden because
      RedisModule_ReplyWithString knows to handle integer encoding of the
      given robj (RedisModuleString).
      
      The PR also introduces a new test to verify the issue is solved.
      a89bf734
  12. 11 Jun, 2020 1 commit
  13. 09 Jun, 2020 1 commit
  14. 03 Jun, 2020 1 commit
  15. 26 May, 2020 1 commit
    • Oran Agra's avatar
      tests: each test client work on a distinct port range · e258a1c0
      Oran Agra authored
      apparently when running tests in parallel (the default of --clients 16),
      there's a chance for two tests to use the same port.
      specifically, one test might shutdown a master and still have the
      replica up, and then another test will re-use the port number of master
      for another master, and then that replica will connect to the master of
      the other test.
      
      this can cause a master to count too many full syncs and fail a test if
      we run the tests with --single integration/psync2 --loop --stop
      
      see Probmem 2 in #7314
      e258a1c0
  16. 20 May, 2020 1 commit
    • Oran Agra's avatar
      fix a rare active defrag edge case bug leading to stagnation · 88d71f47
      Oran Agra authored
      There's a rare case which leads to stagnation in the defragger, causing
      it to keep scanning the keyspace and do nothing (not moving any
      allocation), this happens when all the allocator slabs of a certain bin
      have the same % utilization, but the slab from which new allocations are
      made have a lower utilization.
      
      this commit fixes it by removing the current slab from the overall
      average utilization of the bin, and also eliminate any precision loss in
      the utilization calculation and move the decision about the defrag to
      reside inside jemalloc.
      
      and also add a test that consistently reproduce this issue.
      88d71f47
  17. 14 May, 2020 1 commit
  18. 10 May, 2020 1 commit
    • Yossi Gottlieb's avatar
      TLS: Fix test failures on recent Debian/Ubuntu. · 4d1178cc
      Yossi Gottlieb authored
      Seems like on some systems choosing specific TLS v1/v1.1 versions no
      longer works as expected. Test is reduced for v1.2 now which is still
      good enough to test the mechansim, and matters most anyway.
      4d1178cc
  19. 02 May, 2020 1 commit
    • zhenwei pi's avatar
      Support setcpuaffinity on linux/bsd · 1a0deab2
      zhenwei pi authored
      Currently, there are several types of threads/child processes of a
      redis server. Sometimes we need deeply optimise the performance of
      redis, so we would like to isolate threads/processes.
      
      There were some discussion about cpu affinity cases in the issue:
      https://github.com/antirez/redis/issues/2863
      
      
      
      So implement cpu affinity setting by redis.conf in this patch, then
      we can config server_cpulist/bio_cpulist/aof_rewrite_cpulist/
      bgsave_cpulist by cpu list.
      
      Examples of cpulist in redis.conf:
      server_cpulist 0-7:2      means cpu affinity 0,2,4,6
      bio_cpulist 1,3           means cpu affinity 1,3
      aof_rewrite_cpulist 8-11  means cpu affinity 8,9,10,11
      bgsave_cpulist 1,10-11    means cpu affinity 1,10,11
      
      Test on linux/freebsd, both work fine.
      Signed-off-by: default avatarzhenwei pi <pizhenwei@bytedance.com>
      1a0deab2
  20. 28 Apr, 2020 1 commit
  21. 24 Apr, 2020 1 commit
    • antirez's avatar
      LCS -> STRALGO LCS. · 8a7f255c
      antirez authored
      STRALGO should be a container for mostly read-only string
      algorithms in Redis. The algorithms should have two main
      characteristics:
      
      1. They should be non trivial to compute, and often not part of
      programming language standard libraries.
      2. They should be fast enough that it is a good idea to have optimized C
      implementations.
      
      Next thing I would love to see? A small strings compression algorithm.
      8a7f255c
  22. 22 Apr, 2020 2 commits
  23. 19 Apr, 2020 1 commit
  24. 16 Apr, 2020 1 commit
    • Oran Agra's avatar
      testsuite run the defrag latency test solo · b9fa42a1
      Oran Agra authored
      this test is time sensitive and it sometimes fail to pass below the
      latency threshold, even on strong machines.
      
      this test was the reson we're running just 2 parallel tests in the
      github actions CI, revering this.
      b9fa42a1
  25. 06 Apr, 2020 2 commits
  26. 03 Apr, 2020 1 commit
    • Guy Benoish's avatar
      Try to fix time-sensitive tests in blockonkey.tcl · 1b0d30ae
      Guy Benoish authored
      There is an inherent race between the deferring client and the
      "main" client of the test: While the deferring client issues a blocking
      command, we can't know for sure that by the time the "main" client
      tries to issue another command (Usually one that unblocks the deferring
      client) the deferring client is even blocked...
      For lack of a better choice this commit uses TCL's 'after' in order
      to give some time for the deferring client to issues its blocking
      command before the "main" client does its thing.
      This problem probably exists in many other tests but this commit
      tries to fix blockonkeys.tcl
      1b0d30ae
  27. 02 Apr, 2020 1 commit
  28. 01 Apr, 2020 1 commit
    • Guy Benoish's avatar
      Fix memory corruption in moduleHandleBlockedClients · c4dc5b80
      Guy Benoish authored
      By using a "circular BRPOPLPUSH"-like scenario it was
      possible the get the same client on db->blocking_keys
      twice (See comment in moduleTryServeClientBlockedOnKey)
      
      The fix was actually already implememnted in
      moduleTryServeClientBlockedOnKey but it had a bug:
      the funxction should return 0 or 1 (not OK or ERR)
      
      Other changes:
      1. Added two commands to blockonkeys.c test module (To
         reproduce the case described above)
      2. Simplify blockonkeys.c in order to make testing easier
      3. cast raxSize() to avoid warning with format spec
      c4dc5b80
  29. 31 Mar, 2020 3 commits
  30. 26 Mar, 2020 2 commits
  31. 25 Mar, 2020 1 commit