1. 12 Jan, 2021 2 commits
    • guybe7's avatar
      EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016) · 4b37eb13
      guybe7 authored
      The bug was introduced by #5021 which only attempted avoid EXIST on an
      already expired key from returning 1 on a replica.
      
      Before that commit, dbExists was used instead of
      lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)
      
      Other than that, this commit fixes OBJECT to also come empty handed on
      expired keys in replica.
      
      And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
      the key regardless of it's expired state)
      
      (cherry picked from commit f8ae9917)
      4b37eb13
    • Wang Yuan's avatar
      Disable rehash when redis has child process (#8007) · 3a13c654
      Wang Yuan authored
      In redisFork(), we don't set child pid, so updateDictResizePolicy()
      doesn't take effect, that isn't friendly for copy-on-write.
      
      The bug was introduced this in redis 6.0: 56258c6b
      
      (cherry picked from commit 89c78a98)
      3a13c654
  2. 27 Oct, 2020 24 commits
    • Oran Agra's avatar
      Attempt to fix sporadic test failures due to wait_for_log_messages (#7955) · de0919cd
      Oran Agra authored
      The tests sometimes fail to find a log message.
      Recently i added a print that shows the log files that are searched
      and it shows that the message was in deed there.
      The only reason i can't think of for this seach to fail, is we we
      happened to read an incomplete line, which didn't match our pattern and
      then on the next iteration we would continue reading from the line after
      it.
      
      The fix is to always re-evaluation the previous line.
      
      (cherry picked from commit 4e2e5be2)
      de0919cd
    • Zach Fewtrell's avatar
      fix invalid 'failover' identifier in cluster slave selection test (#7942) · 3988e32f
      Zach Fewtrell authored
      
      (cherry picked from commit ebfa7699)
      3988e32f
    • Qu Chen's avatar
      WATCH no longer ignores keys which have expired for MULTI/EXEC. (#7920) · cebc1f26
      Qu Chen authored
      This wrong behavior was backed by a test, and also documentation, and dates back to 2010.
      But it makes no sense to anyone involved so it was decided to change that.
      
      Note that 20eeddfb (invalidate watch on expire on access) was released in 6.0 RC2
      and 2d1968f8 released in in 6.0.0 GA (invalidate watch when key is evicted).
      both of which do similar changes.
      
      (cherry picked from commit 556acefe)
      cebc1f26
    • Oran Agra's avatar
      improve verbose logging on failed test. print log file lines (#7938) · 56d8ad93
      Oran Agra authored
      
      (cherry picked from commit c96ece9f)
      56d8ad93
    • Yossi Gottlieb's avatar
      Add a --no-latency tests flag. (#7939) · 0103dc09
      Yossi Gottlieb authored
      Useful for running tests on systems which may be way slower than usual.
      
      (cherry picked from commit 843a13e8)
      0103dc09
    • Yossi Gottlieb's avatar
      Fix tests failure on busybox systems. (#7916) · 79e6ab31
      Yossi Gottlieb authored
      
      (cherry picked from commit ef92f507)
      79e6ab31
    • Yossi Gottlieb's avatar
      Modules: fix RM_GetCommandKeys API. (#7901) · 9c475272
      Yossi Gottlieb authored
      This cleans up and simplifies the API by passing the command name as the
      first argument. Previously the command name was specified explicitly,
      but was still included in the argv.
      
      (cherry picked from commit 056a43e1)
      9c475272
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · 6f2c894d
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      (cherry picked from commit adc3183c)
      6f2c894d
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 4147a220
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      
      (cherry picked from commit 0aec98dc)
      4147a220
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 28d1fe67
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      
      (cherry picked from commit 907da058)
      28d1fe67
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 116a2042
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      
      (cherry picked from commit 7d117d75)
      116a2042
    • Madelyn Olson's avatar
      Fixed excessive categories being displayed from acls (#7889) · e7c8002b
      Madelyn Olson authored
      
      (cherry picked from commit 2127f7c8)
      e7c8002b
    • Oran Agra's avatar
      Allow blocked XREAD on a cluster replica (#7881) · bab83457
      Oran Agra authored
      I suppose that it was overlooked, since till recently none of the blocked commands were readonly.
      
      other changes:
      - add test for the above.
      - add better support for additional (and deferring) clients for
        cluster tests
      - improve a test which left the client in MULTI state.
      
      (cherry picked from commit 216c1106)
      bab83457
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · 76f3a63d
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      
      (cherry picked from commit bea40e6a)
      76f3a63d
    • Oran Agra's avatar
      Fix crash in script timeout during AOF loading (#7870) · c80f6219
      Oran Agra authored
      
      (cherry picked from commit dc803d25)
      c80f6219
    • nitaicaro's avatar
      Fixed Tracking test “The other connection is able to get invalidations” (#7871) · a7b95b7c
      nitaicaro authored
      
      
      PROBLEM:
      
      [$rd1 read] reads invalidation messages one by one, so it's never going to see the second invalidation message produced after INCR b, whether or not it exists. Adding another read will block incase no invalidation message is produced.
      
      FIX:
      
      We switch the order of "INCR a" and "INCR b" - now "INCR b" comes first. We still only read the first invalidation message produces. If an invalidation message is wrongly produces for b - then it will be produced before that of a, since "INCR b" comes before "INCR a".
      Co-authored-by: default avatarNitai Caro <caronita@amazon.com>
      (cherry picked from commit 8fb89a57)
      a7b95b7c
    • Oran Agra's avatar
      Fix new obuf-limits tests to work with TLS (#7848) · 120c6b43
      Oran Agra authored
      Also stabilize new shutdown tests on slow machines (valgrind)
      
      (cherry picked from commit 8aa083bd)
      120c6b43
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 757ad7cd
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 57709c4b)
      757ad7cd
    • Wen Hui's avatar
      Add Swapdb Module Event (#7804) · 88441019
      Wen Hui authored
      
      (cherry picked from commit dfe9714c)
      88441019
    • Wang Yuan's avatar
      Remove tmp rdb file in background thread (#7762) · 0bdddd3c
      Wang Yuan authored
      We're already using bg_unlink in several places to delete the rdb file in the background,
      and avoid paying the cost of the deletion from our main thread.
      This commit uses bg_unlink to remove the temporary rdb file in the background too.
      
      However, in case we delete that rdb file just before exiting, we don't actually wait for the
      background thread or the main thread to delete it, and just let the OS clean up after us.
      i.e. we open the file, unlink it and exit with the fd still open.
      
      Furthermore, rdbRemoveTempFile can be called from a thread and was using snprintf which is
      not async-signal-safe, we now use ll2string instead.
      
      (cherry picked from commit b002d2b4)
      0bdddd3c
    • WuYunlong's avatar
      Clarify help text of tcl scripts. (#7798) · 405f041e
      WuYunlong authored
      Before this commit, following command did not show --tls option:
      ./runtest-cluster --help
      ./runtest-sentinel --help
      
      (cherry picked from commit 98c8ac0d)
      405f041e
    • Mykhailo Pylyp's avatar
      aa6b869c
    • Oran Agra's avatar
      fix broken PEXPIREAT test (#7791) · 5106a144
      Oran Agra authored
      This test was nearly always failing on MacOS github actions.
      This is because of bugs in the test that caused it to nearly always run
      all 3 attempts and just look at the last one as the pass/fail creteria.
      
      i.e. the test was nearly always running all 3 attempts and still sometimes
      succeed. this is because the break condition was different than the test
      completion condition.
      
      The reason the test succeeded is because the break condition tested the
      results of all 3 tests (PSETEX/PEXPIRE/PEXPIREAT), but the success check
      at the end was only testing the result of PSETEX.
      
      The reason the PEXPIREAT test nearly always failed is because it was
      getting the current time wrong: getting the current second and loosing
      the sub-section time, so the only chance for it to succeed is if it run
      right when a certain second started.
      
      Because i now get the time from redis, adding another round trip, i
      added another 100ms to the PEXPIRE test to make it less fragile, and
      also added many more attempts.
      
      Adding many more attempts before failure to account for slow platforms,
      github actions and valgrind
      
      (cherry picked from commit ed9bfe22)
      5106a144
    • Oran Agra's avatar
      Squash merging 125 typo/grammar/comment/doc PRs (#7773) · 2ae7f491
      Oran Agra authored
      List of squashed commits or PRs
      ===============================
      
      commit 66801ea
      Author: hwware <wen.hui.ware@gmail.com>
      Date:   Mon Jan 13 00:54:31 2020 -0500
      
          typo fix in acl.c
      
      commit 46f55db
      Author: Itamar Haber <itamar@redislabs.com>
      Date:   Sun Sep 6 18:24:11 2020 +0300
      
          Updates a couple of comments
      
          Specifically:
      
          * RM_AutoMemory completed instead of pointing to docs
          * Updated link to custom type doc
      
      commit 61a2aa0
      Author: xindoo <xindoo@qq.com>
      Date:   Tue Sep 1 19:24:59 2020 +0800
      
          Correct errors in code comments
      
      commit a5871d1
      Author: yz1509 <pro-756@qq.com>
      Date:   Tue Sep 1 18:36:06 2020 +0800
      
          fix typos in module.c
      
      commit 41eede7
      Author: bookug <bookug@qq.com>
      Date:   Sat Aug 15 01:11:33 2020 +0800
      
          docs: fix typos in comments
      
      commit c303c84
      Author: lazy-snail <ws.niu@outlook.com>
      Date:   Fri Aug 7 11:15:44 2020 +0800
      
          fix spelling in redis.conf
      
      commit 1eb76bf
      Author: zhujian <zhujianxyz@gmail.com>...
      2ae7f491
  3. 10 Sep, 2020 14 commits
    • Oran Agra's avatar
      Fix leak in new blockedclient module API test · 18b30467
      Oran Agra authored
      (cherry picked from commit 0e20ad14a6a857cb168b808f94721df19b23dc0c)
      18b30467
    • Yossi Gottlieb's avatar
      Tests: fix oom-score-adj false positives. (#7772) · 0052d519
      Yossi Gottlieb authored
      The key save delay is too short and on certain systems the child process
      is gone before we have a chance to inspect it.
      
      (cherry picked from commit b2a73c40)
      0052d519
    • 杨博东's avatar
      Tests: Add aclfile load and save tests (#7765) · b55b0ea0
      杨博东 authored
      improves test coverage
      
      (cherry picked from commit 0666267d)
      b55b0ea0
    • Roi Lipman's avatar
      RM_ThreadSafeContextTryLock a non-blocking method for acquiring GIL (#7738) · 25e211bc
      Roi Lipman authored
      
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 042189fd)
      25e211bc
    • Yossi Gottlieb's avatar
      Tests: validate CONFIG REWRITE for all params. (#7764) · 635704d4
      Yossi Gottlieb authored
      This is a catch-all test to confirm that that rewrite produces a valid
      output for all parameters and that this process does not introduce
      undesired configuration changes.
      
      (cherry picked from commit a8b72689)
      635704d4
    • Yossi Gottlieb's avatar
      Tests: clean up stale .cli files. (#7768) · 3c8b3945
      Yossi Gottlieb authored
      
      (cherry picked from commit 918abd72)
      3c8b3945
    • Eran Liberty's avatar
      Allow exec with read commands on readonly replica in cluster (#7766) · 84ba2ec8
      Eran Liberty authored
      
      
      There was a bug. Although cluster replicas would allow read commands,
      they would not allow a MULTI-EXEC that's composed solely of read commands.
      Adds tests for coverage.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarEran Liberty <eranl@amazon.com>
      (cherry picked from commit b120366d)
      84ba2ec8
    • Oran Agra's avatar
      handle cur_test for nested tests · 643d3e0b
      Oran Agra authored
      if there are nested tests and nested servers, we need to restore the
      previous value of cur_test when a test exist.
      
      example:
      ```
      test{test 1} {
      	start_server {
      		test{test 1.1 - master only} {
      		}
      		start_server {
      		    test{test 1.2 - with replication} {
                  }
      		}
      	}
      }
      ```
      when `test 1.1 - master only exists`, we're still inside `test 1`
      
      (cherry picked from commit 0a1e7341)
      643d3e0b
    • bodong.ybd's avatar
      Tests: Some fixes for macOS · 781e50d4
      bodong.ybd authored
      1) cur_test: when restart_server, "no such variable" error occurs
        ./runtest --single integration/rdb
        test {client freed during loading}
            SET ::cur_test
            restart_server
              kill_server
                test "Check for memory leaks (pid $pid)"
                SET ::cur_test
                UNSET ::cur_test
            UNSET ::cur_test // This global variable has been unset.
      
      2) `ps --ppid` not available on macOS platform, can be replaced with
      `pgrep -P pid`.
      
      (cherry picked from commit f22fa959)
      781e50d4
    • Oran Agra's avatar
      Fix cluster consistency-check test (#7754) · 57dbd6e9
      Oran Agra authored
      This test was failing from time to time see discussion at the bottom of #7635
      This was probably due to timing, the DEBUG SLEEP executed by redis-cli
      didn't sleep for enough time.
      
      This commit changes:
      1) use SET-ACTIVE-EXPIRE instead of DEBUG SLEEP
      2) reduce many `after` sleeps with retry loops to speed up the test.
      3) add many comment explaining the different steps of the test and
         it's purpose.
      4) config appendonly before populating the volatile keys, so that they'll
         be part of the AOF command stream rather than the preamble RDB portion.
      
      other complications: recently kill_instance switched from SIGKILL to
      SIGTERM, and this would sometimes fail since there was an AOFRW running
      in the background. now we wait for it to end before attempting the kill.
      
      (cherry picked from commit b491d477)
      57dbd6e9
    • Yossi Gottlieb's avatar
      Tests: fix unmonitored servers. (#7756) · e9fef49e
      Yossi Gottlieb authored
      There is an inherent race condition in port allocation for spawned
      servers. If a server fails to start because a port is taken, a new port
      is allocated. This fixes a problem where the logs are not truncated and
      as a result a large number of unmonitored servers are started.
      
      (cherry picked from commit 2df4cb93)
      e9fef49e
    • Oran Agra's avatar
      fix broken cluster/sentinel tests by recent commit (#7752) · a227bc80
      Oran Agra authored
      2b998de4 added a file for stderr to keep valgrind log but i forgot to
      add a similar thing when valgrind isn't being used.
      the result is that `glob */err.txt` fails.
      
      (cherry picked from commit 42ba7a1b)
      a227bc80
    • Oran Agra's avatar
      Improve valgrind support for cluster tests (#7725) · d410dc31
      Oran Agra authored
      - redirect valgrind reports to a dedicated file rather than console
      - try to avoid killing instances with SIGKILL so that we get the memory
        leak report (killing with SIGTERM before resorting to SIGKILL)
      - search for valgrind reports when done, print them and fail the tests
      - add --dont-clean option to keep the logs on exit
      - fix exit error code when crash is found (would have exited with 0)
      
      changes that affect the normal redis test suite:
      - refactor check_valgrind_errors into two functions one to search and
        one to report
      - move the search half into util.tcl to serve the cluster tests too
      - ignore "address range perms" valgrind warnings which seem non relevant.
      
      (cherry picked from commit 2b998de4)
      d410dc31
    • Oran Agra's avatar
      test infra - add durable mode to work around test suite crashing · 41c7c791
      Oran Agra authored
      in some cases a command that returns an error possibly due to a timing
      issue causes the tcl code to crash and thus prevents the rest of the
      tests from running. this adds an option to make the test proceed despite
      the crash.
      maybe it should be the default mode some day.
      
      (cherry picked from commit fe5da2e6)
      41c7c791