1. 11 Mar, 2021 1 commit
  2. 10 Mar, 2021 1 commit
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
  3. 08 Mar, 2021 2 commits
  4. 01 Mar, 2021 3 commits
  5. 28 Feb, 2021 1 commit
    • Viktor Söderqvist's avatar
      Shared reusable client for RM_Call() (#8516) · 6122f1c4
      Viktor Söderqvist authored
      A single client pointer is added in the server struct. This is
      initialized by the first RM_Call() and reused for every subsequent
      RM_Call() except if it's already in use, which means that it's not
      used for (recursive) module calls to modules. For these, a new
      "fake" client is created each time.
      
      Other changes:
      * Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels
        when not needed
      6122f1c4
  6. 24 Feb, 2021 1 commit
  7. 23 Feb, 2021 1 commit
  8. 22 Feb, 2021 1 commit
  9. 21 Feb, 2021 2 commits
    • Huang Zw's avatar
      Client tracking tracking-redir-broken push len is 2 not 3 (#8456) · f687ac0c
      Huang Zw authored
      When redis responds with tracking-redir-broken push message (RESP3),
      it was responding with a broken protocol: an array of 3 elements, but only
      pushes 2 elements.
      
      Some bugs in the test make this pass. Read the push reply
      will consume an extra reply, because the reply length is 3, but there
      are only two elements, so the next reply will be treated as third
      element. So the test is corrected too.
      
      Other changes:
      * checkPrefixCollisionsOrReply success should return 1 instead of -1,
        this bug didn't have any implications.
      * improve client tracking tests to validate more of the response it reads.
      f687ac0c
    • Gnanesh's avatar
      EXPIRE, EXPIREAT, SETEX, GETEX: Return error when expire time overflows (#8287) · 0772098b
      Gnanesh authored
      
      
      Respond with error if expire time overflows from positive to negative of vice versa.
      
      * `SETEX`, `SET EX`, `GETEX` etc would have already error on negative value,
      but now they would also error on overflows (i.e. when the input was positive but
      after the manipulation it becomes negative, which would have passed before)
      * `EXPIRE` and `EXPIREAT` was ok taking negative values (would implicitly delete
      the key), we keep that, but we do error if the user provided a value that changes
      sign when manipulated (except the case of changing sign when `basetime` is added)
      Signed-off-by: default avatarGnanesh <gnaneshkunal@outlook.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      0772098b
  10. 19 Feb, 2021 1 commit
  11. 15 Feb, 2021 3 commits
    • Oran Agra's avatar
      minor test suite cleanup, revive old test (#8497) · fb3457d1
      Oran Agra authored
      There are two tests in other.tcl that were dependant of the sha1 package
      import which meant that they didn't usually run.
      The reason it was like that was that prior to the creation of DEBUG
      DIGEST, the test suite used to have an equivalent function, but that's
      no longer the case and this dependency isn't needed.
      
      The other change is to revert config changes done by the test before the
      test suite continues. can be useful if using `--host` to run multiple
      units against the same server
      fb3457d1
    • Yossi Gottlieb's avatar
      Escape unsafe field name characters in INFO. (#8492) · 141ac8df
      Yossi Gottlieb authored
      Fixes #8489
      141ac8df
    • Viktor Söderqvist's avatar
      Modules: In RM_HashSet, add COUNT_ALL flag and set errno (#8446) · 0bc8c9c8
      Viktor Söderqvist authored
      The added flag affects the return value of RM_HashSet() to include
      the number of inserted fields, in addition to updated and deleted
      fields.
      
      errno is set on errors, tests are added and documentation updated.
      0bc8c9c8
  12. 10 Feb, 2021 1 commit
  13. 08 Feb, 2021 1 commit
    • filipe oliveira's avatar
      [fix] Increasing block on background timeout time to avoid test failure (#8470) · b2351ea0
      filipe oliveira authored
      The test failed from time to time on Github actions.
      We think it's possible that on the module's blocking timeout
      time tracking test, the timeout is happening prior we issue the
      RedisModule_BlockedClientMeasureTimeStart(bc) on the
      background thread. If that is the case one possible solution
      is to increase the timeout.
      Increasing to 200ms to 500ms to see if nightly stops failing.
      b2351ea0
  14. 05 Feb, 2021 2 commits
    • Viktor Söderqvist's avatar
      RM_ZsetRem: Delete key if empty (#8453) · aea6e71e
      Viktor Söderqvist authored
      Without this fix, RM_ZsetRem can leave empty sorted sets which are
      not allowed to exist.
      
      Removing from a sorted set while iterating seems to work (while
      inserting causes failed assetions). RM_ZsetRangeEndReached is
      modified to return 1 if the key doesn't exist, to terminate
      iteration when the last element has been removed.
      aea6e71e
    • sundb's avatar
      RAND* commands: fix risk of OOM panic in hash and zset, use fair random in... · 18ac4197
      sundb authored
      
      RAND* commands: fix risk of OOM panic in hash and zset, use fair random in hash, and add tests for even distribution to all (#8429)
      
      Changes to HRANDFIELD and ZRANDMEMBER:
      * Fix risk of OOM panic when client query a very big negative count (avoid allocating huge temporary buffer).
      * Fix uneven random distribution in HRANDFIELD with negative count (wasn't using dictGetFairRandomKey).
      * Add tests to check an even random distribution (HRANDFIELD, SRANDMEMBER, ZRANDMEMBER).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      18ac4197
  15. 04 Feb, 2021 2 commits
    • Yang Bodong's avatar
      Fix GEOSEARCH tcl test error (#8451) · b7b23a0f
      Yang Bodong authored
      Issue with new test due to longitude wraparound.
      b7b23a0f
    • Yang Bodong's avatar
      GEOSEARCH bybox bug fixes and new fuzzy tester (#8445) · ded1655d
      Yang Bodong authored
      
      
      Fix errors of GEOSEARCH bybox search due to:
      1. projection of the box to a trapezoid (when the meter box is converted to long / lat it's no longer a box).
      2. width and height mismatch
      
      Changes:
      - New GEOSEARCH point in rectangle algorithm
      - Fix GEOSEARCH bybox width and height mismatch bug
      - Add GEOSEARCH bybox testing to the existing "GEOADD + GEORANGE randomized test"
      - Add new fuzzy test to stress test the bybox corners and edges
      - Add some tests for edge cases of the bybox algorithm
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ded1655d
  16. 29 Jan, 2021 2 commits
    • filipe oliveira's avatar
      Enabled background and reply time tracking on blocked on keys/blocked on... · f0c5052a
      filipe oliveira authored
      Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
      
      This commit enables tracking time of the background tasks and on replies,
      opening the door for properly tracking commands that rely on blocking / background
       work via the slowlog, latency history, and commandstats. 
      
      Some notes:
      - The time spent blocked waiting for key changes, or blocked on synchronous
        replication is not accounted for. 
      
      - **This commit does not affect latency tracking of commands that are non-blocking
        or do not have background work.** ( meaning that it all stays the same with exception to
        `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
        on background threads ). 
      
      -  Specifically for latency history command we've added a new event class named
        `command-unblocking` that will enable latency monitoring on commands that spawn
        background threads to do the work.
      
      - For blocking commands we're now considering the total time of a command as the
        time spent on call() + the time spent on replying when unblocked.
      
      - For Modules commands that rely on background threads we're now considering the
        total time of a command as the time spent on call (main thread) + the time spent on
        the background thread ( if marked within `RedisModule_MeasureTimeStart()` and
        `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread)
      
      To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on
      a module that blocks the client and sleeps on the background for a given time. 
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout
      - check blocked command with multiple calls RedisModule_MeasureTimeStart()  is tracking the total background time
      - check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
      f0c5052a
    • Yang Bodong's avatar
      Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297) · b9a0500f
      Yang Bodong authored
      
      
      New commands:
      `HRANDFIELD [<count> [WITHVALUES]]`
      `ZRANDMEMBER [<count> [WITHSCORES]]`
      Algorithms are similar to the one in SRANDMEMBER.
      
      Both return a simple bulk response when no arguments are given, and an array otherwise.
      In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
      note: in all 3 commands, the only option that also provides random order is the one with negative count.
      
      Changes to SRANDMEMBER
      * Optimization when count is 1, we can use the more efficient algorithm of non-unique random
      * optimization: work with sds strings rather than robj
      
      Other changes:
      * zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
        case the buffer is too small)
      * Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
        case 4) and by accident used a negative count
      Co-authored-by: default avatarxinluton <xinluton@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b9a0500f
  17. 28 Jan, 2021 3 commits
    • Yossi Gottlieb's avatar
      Add proc-title-template option. (#8397) · 4bb5ccbe
      Yossi Gottlieb authored
      Make it possible to customize the process title, i.e. include custom
      strings, immutable configuration like port, tls-port, unix socket name,
      etc.
      4bb5ccbe
    • Viktor Söderqvist's avatar
      Add modules API for streams (#8288) · 4355145a
      Viktor Söderqvist authored
      APIs added for these stream operations: add, delete, iterate and
      trim (by ID or maxlength). The functions are prefixed by RM_Stream.
      
      * RM_StreamAdd
      * RM_StreamDelete
      * RM_StreamIteratorStart
      * RM_StreamIteratorStop
      * RM_StreamIteratorNextID
      * RM_StreamIteratorNextField
      * RM_StreamIteratorDelete
      * RM_StreamTrimByLength
      * RM_StreamTrimByID
      
      The type RedisModuleStreamID is added and functions for converting
      from and to RedisModuleString.
      
      * RM_CreateStringFromStreamID
      * RM_StringToStreamID
      
      Whenever the stream functions return REDISMODULE_ERR, errno is set to
      provide additional error information.
      
      Refactoring: The zset iterator fields in the RedisModuleKey struct
      are wrapped in a union, to allow the same space to be used for type-
      specific info for streams and allow future use for other key types.
      4355145a
    • Z. Liu's avatar
      Add 'set-proc-title' config so that this mechanism can be disabled (#3623) · 17b34c73
      Z. Liu authored
      
      
      if option `set-proc-title' is no, then do nothing for proc title.
      
      The reason has been explained long ago, see following:
      
      We update redis to 2.8.8, then found there are some side effect when
      redis always change the process title.
      
      We run several slave instance on one computer, and all these salves
      listen on unix socket only, then ps will show:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0
      
      for redis 2.6 the output of ps is like following:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf
      
      Later is more informational in our case. The situation
      is worse when we manage the config and process running
      state by salt. Salt check the process by running "ps |
      grep SIG" (for Gentoo System) to check the running
      state, where SIG is the string to search for when
      looking for the service process with ps. Previously, we
      define sig as "/usr/sbin/redis-server
      /etc/redis/a.conf". Since the ps output is identical for
      our case, so we have no way to check the state of
      specified redis instance.
      
      So, for our case, we prefer the old behavior, i.e, do
      not change the process title for the main redis process.
      Or add an option such as "set-proc-title [yes|no]" to
      control this behavior.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      17b34c73
  18. 27 Jan, 2021 1 commit
    • Raghav Muddur's avatar
      GETEX, GETDEL and SET PXAT/EXAT (#8327) · 0367a808
      Raghav Muddur authored
      This commit introduces two new command and two options for an existing command
      
      GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
      [PXAT milliseconds-timestamp]
      
      The getexCommand() function implements extended options and variants of the GET
      command. Unlike GET command this command is not read-only. Only one of the options
      can be used at a given time.
      
      1. PERSIST removes any TTL associated with the key.
      2. EX Set expiry TTL in seconds.
      3. PX Set expiry TTL in milliseconds.
      4. EXAT Same like EX instead of specifying the number of seconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      5. PXAT Same like PX instead of specifying the number of milliseconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      
      Command would return either the bulk string, error or nil.
      
      GETDEL <key>
      Would delete the key after getting.
      
      SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] ...
      0367a808
  19. 26 Jan, 2021 1 commit
  20. 25 Jan, 2021 1 commit
    • Oran Agra's avatar
      Fix rare test failures due to repl-ping-replica-period (#8393) · 437e2583
      Oran Agra authored
      some tests use attach_to_replication_stream to watch what's propagated
      to replicas, but in some cases the periodic ping may slip in and fail
      the test.
      we disable that ping by setting the period to once an hour (tests should
      not run for that long).
      
      other change is so that the next time this oom-score-adj test fails,
      we'll see the value (assert_equals prints it)
      437e2583
  21. 23 Jan, 2021 1 commit
    • Oran Agra's avatar
      Fix recent test failures (#8386) · f2258915
      Oran Agra authored
      1. Valgrind leak in a recent change in a module api test
      2. Increase treshold of a RESTORE TTL test
      3. Change assertions to use assert_range which prints the values
      f2258915
  22. 22 Jan, 2021 1 commit
    • Viktor Söderqvist's avatar
      Test that module can wake up module blocked on non-empty list key (#8382) · 9c148310
      Viktor Söderqvist authored
      BLPOP and other blocking list commands can only block on empty keys
      and LPUSH only wakes up clients when the list is created.
      
      Using the module API, it's possible to block on a non-empty key.
      Unblocking a client blocked on a non-empty list (or zset) can only
      be done using RedisModule_SignalKeyAsReady(). This commit tests it.
      9c148310
  23. 20 Jan, 2021 1 commit
    • guybe7's avatar
      Fix misleading module test (#8366) · 5a77d015
      guybe7 authored
      the test was misleading because the module would actually woke up on a wrong type and
      re-blocked, while the test name suggests the module doesn't not wake up at all on a wrong type..
      
      i changed the name of the test + added verification that indeed the module wakes up and gets
      re-blocked after it understand it's the wrong type
      5a77d015
  24. 19 Jan, 2021 2 commits
    • Oran Agra's avatar
      Add tests to make sure that relative EXPIRE is propagated to replicas (#8357) · a29aec9a
      Oran Agra authored
      This commit adds tests to make sure that relative and absolute expire commands
      are propagated as is to replicas and stop any future attempt to change that without
      a proper discussion. see #8327 and #5171
      
      Additionally it slightly improve the AOF test that tests the opposite (always
      propagating absolute times), by covering more commands, and shaving 2
      seconds from the test time.
      a29aec9a
    • Viktor Söderqvist's avatar
      Bugfix: Make modules blocked on keys unblock on commands like LPUSH (#8356) · 4985c11b
      Viktor Söderqvist authored
      This was a regression from #7625 (only in 6.2 RC2).
      
      This makes it possible again to implement blocking list and zset
      commands using the modules API.
      
      This commit also includes a test case for the reverse: A module
      unblocks a client blocked on BLPOP by inserting elements using
      RedisModule_ListPush(). This already works, but it was untested.
      4985c11b
  25. 17 Jan, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add io-thread daily CI tests. (#8232) · 522d9360
      Yossi Gottlieb authored
      This adds basic coverage to IO threads by running the cluster and few selected Redis test suite tests with the IO threads enabled.
      
      Also provides some necessary additional improvements to the test suite:
      
      * Add --config to sentinel/cluster tests for arbitrary configuration.
      * Fix --tags whitelisting which was broken.
      * Add a `network` tag to some tests that are more network intensive. This is work in progress and more tests should be properly tagged in the future.
      522d9360
  26. 15 Jan, 2021 1 commit
    • Yang Bodong's avatar
      Add lazyfree-lazy-user-flush config to control default behavior of... · 294f93af
      Yang Bodong authored
      Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258)
      
      * Adds ASYNC and SYNC arguments to SCRIPT FLUSH
      * Adds SYNC argument to FLUSHDB and FLUSHALL
      * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH.
      
      the new behavior is as follows:
      * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the
        value of lazyfree-lazy-user-flush.
      * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner.
      * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
      294f93af
  27. 12 Jan, 2021 1 commit
    • Madelyn Olson's avatar
      Fix issues in wait test (#8310) · b24b4903
      Madelyn Olson authored
      This fixes three issues:
      1.  Using debug SLEEP was impacting the subsequent test, and causing it to pass reliably even though it should have failed. There was exactly 5 seconds of artificial pause (after 1000, wait 3000, wait 1000) between the debug sleep 5 and when we needed to unblock the client in the subsequent test. Now the test properly makes sure the client is unblocked, and the subsequent test is fixed.
      2. Minor, the client pause types were using & comparisons instead of ==, since it was previously a flag.
      3. Test is faster now that some of the hand wavy time is removed. 
      b24b4903
  28. 08 Jan, 2021 1 commit