1. 09 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve test suite to handle external servers better. (#9033) · 8a86bca5
      Yossi Gottlieb authored
      This commit revives the improves the ability to run the test suite against
      external servers, instead of launching and managing `redis-server` processes as
      part of the test fixture.
      
      This capability existed in the past, using the `--host` and `--port` options.
      However, it was quite limited and mostly useful when running a specific tests.
      Attempting to run larger chunks of the test suite experienced many issues:
      
      * Many tests depend on being able to start and control `redis-server` themselves,
      and there's no clear distinction between external server compatible and other
      tests.
      * Cluster mode is not supported (resulting with `CROSSSLOT` errors).
      
      This PR cleans up many things and makes it possible to run the entire test suite
      against an external server. It also provides more fine grained controls to
      handle cases where the external server supports a subset of the Redis commands,
      limited number of databases, cluster mode, etc.
      
      The tests directory now contains a `README.md` file that describes how this
      works.
      
      This commit also includes additional cleanups and fixes:
      
      * Tests can now be tagged.
      * Tag-based selection is now unified across `start_server`, `tags` and `test`.
      * More information is provided about skipped or ignored tests.
      * Repeated patterns in tests have been extracted to common procedures, both at a
        global level and on a per-test file basis.
      * Cleaned up some cases where test setup was based on a previous test executing
        (a major anti-pattern that repeats itself in many places).
      * Cleaned up some cases where test teardown was not part of a test (in the
        future we should have dedicated teardown code that executes even when tests
        fail).
      * Fixed some tests that were flaky running on external servers.
      8a86bca5
  2. 07 Jun, 2021 2 commits
  3. 01 Jun, 2021 2 commits
    • Oran Agra's avatar
      7cb42c9c
    • Oran Agra's avatar
      Improve new time sensitive pexpireat propagation test (#9010) · ae67539c
      Oran Agra authored
      The test that was merged yesterday fails with valgrind and freebsd CI
      that are too slow, and 10 seconds apparently passed between the time the
      command was sent to redis and the time it was actually executed.
      
      ```
      *** [err]: All TTL in commands are propagated as absolute timestamp in replication stream in tests/unit/expire.tcl
      Expected 'del a' to match 'set foo1 bar PXAT *' (context: type source line 778 file /home/runner/work/redis/redis/tests/test_helper.tcl cmd {assert_match [lindex $patterns $j] [read_from_replication_stream $s]} proc ::assert_replication_stream level 1)
      ```
      ae67539c
  4. 30 May, 2021 1 commit
    • ny0312's avatar
      Always replicate time-to-live(TTL) as absolute timestamps in milliseconds (#8474) · 53d1acd5
      ny0312 authored
      Till now, on replica full-sync we used to transfer absolute time for TTL,
      however when a command arrived (EXPIRE or EXPIREAT),
      we used to propagate it as is to replicas (possibly with relative time),
      but always translate it to EXPIREAT (absolute time) to AOF.
      
      This commit changes that and will always use absolute time for propagation.
      see discussion in #8433
      
      Furthermore, we Introduce new commands: `EXPIRETIME/PEXPIRETIME`
      that allow extracting the absolute TTL time from a key.
      53d1acd5
  5. 20 May, 2021 1 commit
    • YaacovHazan's avatar
      stabilize tests that involved with load handlers (#8967) · 32a2584e
      YaacovHazan authored
      When test stop 'load handler' by killing the process that generating the load,
      some commands that already in the input buffer, still might be processed by the server.
      This may cause some instability in tests, that count on that no more commands
      processed after we stop the `load handler'
      
      In this commit, new proc 'wait_load_handlers_disconnected' added, to verify that no more
      cammands from any 'load handler' prossesed, by checking that the clients who
      genreate the load is disconnceted.
      
      Also, replacing check of dbsize with wait_for_ofs_sync before comparing debug digest, as
      it would fail in case the last key the workload wrote was an overridden key (not a new one).
      
      Affected tests
      Race fix:
      - failover command to specific replica works
      - Connect multiple replicas at the same time (issue #141), master diskless=$mdl, replica diskless=$sdl
      - AOF rewrite during write load: RDB preamble=$rdbpre
      
      Cleanup and speedup:
      - Test replication with blocking lists and sorted sets operations
      - Test replication with parallel clients writing in different DBs
      - Test replication partial resync: $descr (diskless: $mdl, $sdl, reconnect: $reconnect
      32a2584e
  6. 19 May, 2021 2 commits
    • Madelyn Olson's avatar
      Hide migrate command from slowlog if they include auth (#8859) · a59e75a4
      Madelyn Olson authored
      Redact commands that include sensitive data from slowlog and monitor
      a59e75a4
    • Oran Agra's avatar
      Fix race in new lazyfree test (#8965) · d67e66de
      Oran Agra authored
      I recently saw this failure:
      [err]: lazy free a stream with all types of metadata in tests/unit/lazyfree.tcl
      Expected '2' to be equal to '1' (context: type eval line 23 cmd {assert_equal [s lazyfreed_objects] 1} proc ::test)
      
      The only explanation for such a thing is that the async flushdb wasn't
      done before we did the resetstat
      d67e66de
  7. 18 May, 2021 1 commit
  8. 17 May, 2021 1 commit
  9. 13 May, 2021 1 commit
  10. 04 May, 2021 1 commit
  11. 27 Apr, 2021 1 commit
  12. 20 Apr, 2021 1 commit
  13. 19 Apr, 2021 4 commits
    • Hanna Fadida's avatar
      Modules: adding a module type for key space notification (#8759) · 53a4d6c3
      Hanna Fadida authored
      Adding a new type mask ​for key space notification, REDISMODULE_NOTIFY_MODULE, to enable unique notifications from commands on REDISMODULE_KEYTYPE_MODULE type keys (which is currently unsupported).
      
      Modules can subscribe to a module key keyspace notification by RM_SubscribeToKeyspaceEvents,
      and clients by notify-keyspace-events of redis.conf or via the CONFIG SET, with the characters 'd' or 'A' 
      (REDISMODULE_NOTIFY_MODULE type mask is part of the '**A**ll' notation for key space notifications).
      
      Refactor: move some pubsub test infra from pubsub.tcl to util.tcl to be re-used by other tests.
      53a4d6c3
    • guybe7's avatar
      Modules: Replicate lazy-expire even if replication is not allowed (#8816) · f40ca9cb
      guybe7 authored
      Before this commit using RM_Call without "!" could cause the master
      to lazy-expire a key (delete it) but without replicating to replicas.
      This could cause the replica's memory usage to gradually grow and
      could also cause consistency issues if the master and replica have
      a clock diff.
      This bug was introduced in #8617
      
      Added a test which demonstrates that scenario.
      f40ca9cb
    • Harkrishn Patro's avatar
      ACL channels permission handling for save/load scenario. (#8794) · 7a3d1487
      Harkrishn Patro authored
      
      
      In the initial release of Redis 6.2 setting a user to only allow pubsub access to
      a specific channel, and doing ACL SAVE, resulted in an assertion when
      ACL LOAD was used. This was later changed by #8723 (not yet released),
      but still not properly resolved (now it errors instead of crash).
      
      The problem is that the server that generates an ACL file, doesn't know what
      would be the setting of the acl-pubsub-default config in the server that will load it.
      so ACL SAVE needs to always start with resetchannels directive.
      
      This should still be compatible with old acl files (from redis 6.0), and ones from earlier
      versions of 6.2 that didn't mess with channels.
      Co-authored-by: default avatarHarkrishn Patro <harkrisp@amazon.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      7a3d1487
    • sundb's avatar
      Fix ouput buffer limit test (#8803) · 3a955d9a
      sundb authored
      The tail size of c->reply is 16kb, but in the test only publish a
      few chars each time, due to a change in #8699, the obuf limit
      is now checked a new memory allocation is made, so this test
      would have sometimes failed to trigger a soft limit disconnection
      in time.
      
      The solution is to write bigger payloads to the output buffer, but
      still limit their rate (not more than 100k/s).
      3a955d9a
  14. 14 Apr, 2021 1 commit
  15. 13 Apr, 2021 1 commit
  16. 07 Apr, 2021 1 commit
  17. 05 Apr, 2021 1 commit
    • Huang Zhw's avatar
      Fix "default" and overwritten / reset users will not have pubsub channels... · 3b74b550
      Huang Zhw authored
      Fix "default" and overwritten / reset users will not have pubsub channels permissions by default. (#8723)
      
      Background:
      Redis 6.2 added ACL control for pubsub channels (#7993), which were supposed
      to be permissive by default to retain compatibility with redis 6.0 ACL. 
      But due to a bug, only newly created users got this `acl-pubsub-default` applied,
      while overwritten (updated) users got reset to `resetchannels` (denied).
      
      Since the "default" user exists before loading the config file,
      any ACL change to it, results in an update / overwrite.
      
      So when a "default" user is loaded from config file or include ACL
      file with no channels related rules, the user will not have any
      permissions to any channels. But other users will have default
      permissions to any channels.
      
      When upgraded from 6.0 with config rewrite, this will lead to
      "default" user channels permissions lost.
      When users are loaded from include file, then call "acl load", users
      will also lost channels permissions.
      
      Similarly, the `reset` ACL rule, would have reset the user to be denied
      access to any channels, ignoring `acl-pubsub-default` and breaking
      compatibility with redis 6.0.
      
      The implication of this fix is that it regains compatibility with redis 6.0,
      but breaks compatibility with redis 6.2.0 and 2.0.1. e.g. after the upgrade,
      the default user will regain access to pubsub channels.
      
      Other changes:
      Additionally this commit rename server.acl_pubusub_default to
      server.acl_pubsub_default and fix typo in acl tests.
      3b74b550
  18. 01 Apr, 2021 3 commits
    • Valentino Geron's avatar
      Fix XAUTOCLAIM response to return the next available id as the cursor (#8725) · 44d8b039
      Valentino Geron authored
      This command used to return the last scanned entry id as the cursor,
      instead of the next one to be scanned.
      so in the next call, the user could / should have sent `(cursor` and not
      just `cursor` if he wanted to avoid scanning the same record twice.
      
      Scanning the record twice would look odd if someone is checking what
      exactly was scanned, but it also has a side effect of incrementing the
      delivery count twice.
      44d8b039
    • guybe7's avatar
      zsetAdd: Fix wrong reply in case of INCR and GT/LT (#8717) · 843f769b
      guybe7 authored
      If GT/LT fails the operation we need to reply with
      nill (like failure due to NX).
      
      Other changes:
      Add the missing $encoding suffix to many zset tests
      
      Note: there's a behavior change just in case of INCR + GT/LT that fails.
      The old code was replying with the wrong (rejected) score, and now it'll reply with nil.
      
      Note that that's anyway a corner case so this "behavior change" shouldn't have too much affect.
      Using GT/LT with INCR has a predictable result even before we run the command
      (INCR GT will only only / always fail if the increment is negative).
      843f769b
    • sundb's avatar
      Use chi-square for random distributivity verification in test (#8709) · 569a3f45
      sundb authored
      Problem:
      Currently, when performing random distribution verification, we determine
      the probability of each element occurring in the sum, but the probability is
      only an estimate, these tests had rare sporadic failures, and we cannot verify
      what the probability of failure will be.
      
      Solution:
      Using the chi-square distribution instead of the original random distribution
      validation makes the test more reasonable and easier to find problems.
      569a3f45
  19. 30 Mar, 2021 1 commit
    • JunhuaY's avatar
      re-fix config rewrite for empty save directive (#8722) · 28375ff6
      JunhuaY authored
      the bug was also discussed in #8716, and was solved in #8719, but incompletely:
      when the server is started, and the save option is default, if you issue the " config set save "" "
      to change the save option, and then issue the “config rewrite” command, the " save "" " won't be saved.
      28375ff6
  20. 29 Mar, 2021 2 commits
  21. 26 Mar, 2021 1 commit
    • Huang Zhw's avatar
      make processCommand check publish channel permissions. (#8534) · e138698e
      Huang Zhw authored
      Add publish channel permissions check in processCommand.
      
      processCommand didn't check publish channel permissions, so we can
      queue a publish command in a transaction. But when exec the transaction,
      it will fail with -NOPERM.
      
      We also union keys/commands/channels permissions check togegher in
      ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
      publishCommand/subscribeCommand/psubscribeCommand. Always 
      check permissions in processCommand/execCommand/
      luaRedisGenericCommand.
      e138698e
  22. 25 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix SLOWLOG for blocked commands (#8632) · 497351ad
      Oran Agra authored
      * SLOWLOG didn't record anything for blocked commands because the client
        was reset and argv was already empty. there was a fix for this issue
        specifically for modules, now it works for all blocked clients.
      * The original command argv (before being re-written) was also reset
        before adding the slowlog on behalf of the blocked command.
      * Latency monitor is now updated regardless of the slowlog flags of the
        command or its execution (their purpose is to hide sensitive info from
        the slowlog, not hide the fact the latency happened).
      * Latency monitor now uses real_cmd rather than c->cmd (which may be
        different if the command got re-written, e.g. GEOADD)
      
      Changes:
      * Unify shared code between slowlog insertion in call() and
        updateStatsOnUnblock(), hopefully prevent future bugs from happening
        due to the later being overlooked.
      * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
        processing.
      * Add a test for SLOWLOG and BLPOP
      
      Notes:
      - real_cmd == c->lastcmd, except inside MULTI and Lua.
      - blocked commands never happen in these cases (MULTI / Lua)
      - real_cmd == c->cmd, except for when the command is rewritten (e.g.
        GEOADD)
      - blocked commands (currently) are never rewritten
      - other than the command's CLIENT_PREVENT_LOGGING, and the
        execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
        avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
        be off when executed from execCommand that runs from an AOF)
      497351ad
  23. 24 Mar, 2021 1 commit
  24. 22 Mar, 2021 1 commit
  25. 17 Mar, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix script kill to work also on scripts that use pcall (#8661) · 9ae4f5c7
      Meir Shpilraien (Spielrein) authored
      pcall function runs another LUA function in protected mode, this means
      that any error will be caught by this function and will not stop the LUA
      execution. The script kill mechanism uses error to stop the running script.
      Scripts that uses pcall can catch the error raise by the script kill mechanism,
      this will cause a script like this to be unkillable:
      
      local f = function()
              while 1 do
                      redis.call('ping')
              end
      end
      while 1 do
              pcall(f)
      end
      
      The fix is, when we want to kill the script, we set the hook function to be invoked 
      after each line. This will promise that the execution will get another
      error before it is able to enter the pcall function again.
      9ae4f5c7
  26. 16 Mar, 2021 1 commit
  27. 15 Mar, 2021 1 commit
    • guybe7's avatar
      Missing EXEC on modules propagation after failed EVAL execution (#8654) · dba33a94
      guybe7 authored
      1. moduleReplicateMultiIfNeeded should use server.in_eval like
         moduleHandlePropagationAfterCommandCallback
      2. server.in_eval could have been set to 1 and not reset back
         to 0 (a lot of missed early-exits after in_eval is already 1)
      
      Note: The new assertions in processCommand cover (2) and I added
      two module tests to cover (1)
      
      Implications:
      If an EVAL that failed (and thus left server.in_eval=1) runs before a module
      command that replicates, the replication stream will contain MULTI (because
      moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL
      at this point) but not EXEC (because server.in_eval==1)
      This only affects modules as module.c the only user of server.in_eval.
      
      Affects versions 6.2.0, 6.2.1
      dba33a94
  28. 11 Mar, 2021 1 commit
  29. 10 Mar, 2021 1 commit
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
  30. 08 Mar, 2021 2 commits