1. 22 Jun, 2021 1 commit
    • Evan's avatar
      modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718) · 1ccf2ca2
      Evan authored
      Previously, passing 0 for newlen would not truncate the string at all.
      This adds handling of this case, freeing the old string and creating a new empty string.
      
      Other changes:
      - Move `src/modules/testmodule.c` to `tests/modules/basics.c`
      - Introduce that basic test into the test suite
      - Add tests to cover StringTruncate
      - Add `test-modules` build target for the main makefile
      - Extend `distclean` build target to clean modules too
      1ccf2ca2
  2. 16 Jun, 2021 1 commit
  3. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  4. 01 Jun, 2021 1 commit
  5. 19 Apr, 2021 2 commits
    • Hanna Fadida's avatar
      Modules: adding a module type for key space notification (#8759) · 53a4d6c3
      Hanna Fadida authored
      Adding a new type mask ​for key space notification, REDISMODULE_NOTIFY_MODULE, to enable unique notifications from commands on REDISMODULE_KEYTYPE_MODULE type keys (which is currently unsupported).
      
      Modules can subscribe to a module key keyspace notification by RM_SubscribeToKeyspaceEvents,
      and clients by notify-keyspace-events of redis.conf or via the CONFIG SET, with the characters 'd' or 'A' 
      (REDISMODULE_NOTIFY_MODULE type mask is part of the '**A**ll' notation for key space notifications).
      
      Refactor: move some pubsub test infra from pubsub.tcl to util.tcl to be re-used by other tests.
      53a4d6c3
    • guybe7's avatar
      Modules: Replicate lazy-expire even if replication is not allowed (#8816) · f40ca9cb
      guybe7 authored
      Before this commit using RM_Call without "!" could cause the master
      to lazy-expire a key (delete it) but without replicating to replicas.
      This could cause the replica's memory usage to gradually grow and
      could also cause consistency issues if the master and replica have
      a clock diff.
      This bug was introduced in #8617
      
      Added a test which demonstrates that scenario.
      f40ca9cb
  6. 15 Mar, 2021 1 commit
    • guybe7's avatar
      Missing EXEC on modules propagation after failed EVAL execution (#8654) · dba33a94
      guybe7 authored
      1. moduleReplicateMultiIfNeeded should use server.in_eval like
         moduleHandlePropagationAfterCommandCallback
      2. server.in_eval could have been set to 1 and not reset back
         to 0 (a lot of missed early-exits after in_eval is already 1)
      
      Note: The new assertions in processCommand cover (2) and I added
      two module tests to cover (1)
      
      Implications:
      If an EVAL that failed (and thus left server.in_eval=1) runs before a module
      command that replicates, the replication stream will contain MULTI (because
      moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL
      at this point) but not EXEC (because server.in_eval==1)
      This only affects modules as module.c the only user of server.in_eval.
      
      Affects versions 6.2.0, 6.2.1
      dba33a94
  7. 10 Mar, 2021 1 commit
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
  8. 01 Mar, 2021 1 commit
  9. 28 Feb, 2021 1 commit
    • Viktor Söderqvist's avatar
      Shared reusable client for RM_Call() (#8516) · 6122f1c4
      Viktor Söderqvist authored
      A single client pointer is added in the server struct. This is
      initialized by the first RM_Call() and reused for every subsequent
      RM_Call() except if it's already in use, which means that it's not
      used for (recursive) module calls to modules. For these, a new
      "fake" client is created each time.
      
      Other changes:
      * Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels
        when not needed
      6122f1c4
  10. 15 Feb, 2021 2 commits
  11. 10 Feb, 2021 1 commit
  12. 08 Feb, 2021 1 commit
    • filipe oliveira's avatar
      [fix] Increasing block on background timeout time to avoid test failure (#8470) · b2351ea0
      filipe oliveira authored
      The test failed from time to time on Github actions.
      We think it's possible that on the module's blocking timeout
      time tracking test, the timeout is happening prior we issue the
      RedisModule_BlockedClientMeasureTimeStart(bc) on the
      background thread. If that is the case one possible solution
      is to increase the timeout.
      Increasing to 200ms to 500ms to see if nightly stops failing.
      b2351ea0
  13. 05 Feb, 2021 1 commit
    • Viktor Söderqvist's avatar
      RM_ZsetRem: Delete key if empty (#8453) · aea6e71e
      Viktor Söderqvist authored
      Without this fix, RM_ZsetRem can leave empty sorted sets which are
      not allowed to exist.
      
      Removing from a sorted set while iterating seems to work (while
      inserting causes failed assetions). RM_ZsetRangeEndReached is
      modified to return 1 if the key doesn't exist, to terminate
      iteration when the last element has been removed.
      aea6e71e
  14. 29 Jan, 2021 1 commit
    • filipe oliveira's avatar
      Enabled background and reply time tracking on blocked on keys/blocked on... · f0c5052a
      filipe oliveira authored
      Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
      
      This commit enables tracking time of the background tasks and on replies,
      opening the door for properly tracking commands that rely on blocking / background
       work via the slowlog, latency history, and commandstats. 
      
      Some notes:
      - The time spent blocked waiting for key changes, or blocked on synchronous
        replication is not accounted for. 
      
      - **This commit does not affect latency tracking of commands that are non-blocking
        or do not have background work.** ( meaning that it all stays the same with exception to
        `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
        on background threads ). 
      
      -  Specifically for latency history command we've added a new event class named
        `command-unblocking` that will enable latency monitoring on commands that spawn
        background threads to do the work.
      
      - For blocking commands we're now considering the total time of a comman...
      f0c5052a
  15. 28 Jan, 2021 1 commit
    • Viktor Söderqvist's avatar
      Add modules API for streams (#8288) · 4355145a
      Viktor Söderqvist authored
      APIs added for these stream operations: add, delete, iterate and
      trim (by ID or maxlength). The functions are prefixed by RM_Stream.
      
      * RM_StreamAdd
      * RM_StreamDelete
      * RM_StreamIteratorStart
      * RM_StreamIteratorStop
      * RM_StreamIteratorNextID
      * RM_StreamIteratorNextField
      * RM_StreamIteratorDelete
      * RM_StreamTrimByLength
      * RM_StreamTrimByID
      
      The type RedisModuleStreamID is added and functions for converting
      from and to RedisModuleString.
      
      * RM_CreateStringFromStreamID
      * RM_StringToStreamID
      
      Whenever the stream functions return REDISMODULE_ERR, errno is set to
      provide additional error information.
      
      Refactoring: The zset iterator fields in the RedisModuleKey struct
      are wrapped in a union, to allow the same space to be used for type-
      specific info for streams and allow future use for other key types.
      4355145a
  16. 22 Jan, 2021 1 commit
    • Viktor Söderqvist's avatar
      Test that module can wake up module blocked on non-empty list key (#8382) · 9c148310
      Viktor Söderqvist authored
      BLPOP and other blocking list commands can only block on empty keys
      and LPUSH only wakes up clients when the list is created.
      
      Using the module API, it's possible to block on a non-empty key.
      Unblocking a client blocked on a non-empty list (or zset) can only
      be done using RedisModule_SignalKeyAsReady(). This commit tests it.
      9c148310
  17. 20 Jan, 2021 1 commit
    • guybe7's avatar
      Fix misleading module test (#8366) · 5a77d015
      guybe7 authored
      the test was misleading because the module would actually woke up on a wrong type and
      re-blocked, while the test name suggests the module doesn't not wake up at all on a wrong type..
      
      i changed the name of the test + added verification that indeed the module wakes up and gets
      re-blocked after it understand it's the wrong type
      5a77d015
  18. 19 Jan, 2021 1 commit
    • Viktor Söderqvist's avatar
      Bugfix: Make modules blocked on keys unblock on commands like LPUSH (#8356) · 4985c11b
      Viktor Söderqvist authored
      This was a regression from #7625 (only in 6.2 RC2).
      
      This makes it possible again to implement blocking list and zset
      commands using the modules API.
      
      This commit also includes a test case for the reverse: A module
      unblocks a client blocked on BLPOP by inserting elements using
      RedisModule_ListPush(). This already works, but it was untested.
      4985c11b
  19. 22 Dec, 2020 1 commit
    • Oran Agra's avatar
      Remove read-only flag from non-keyspace cmds, different approach for EXEC to... · 411c18bb
      Oran Agra authored
      Remove read-only flag from non-keyspace cmds, different approach for EXEC to propagate MULTI (#8216)
      
      In the distant history there was only the read flag for commands, and whatever
      command that didn't have the read flag was a write one.
      Then we added the write flag, but some portions of the code still used !read
      Also some commands that don't work on the keyspace at all, still have the read
      flag.
      
      Changes in this commit:
      1. remove the read-only flag from TIME, ECHO, ROLE and LASTSAVE
      
      2. EXEC command used to decides if it should propagate a MULTI by looking at
         the command flags (!read & !admin).
         When i was about to change it to look at the write flag instead, i realized
         that this would cause it not to propagate a MULTI for PUBLISH, EVAL, and
         SCRIPT, all 3 are not marked as either a read command or a write one (as
         they should), but all 3 are calling forceCommandPropagation.
      
         So instead of introducing a new flag to denote a command that "writes" but
         not into the keyspace, and still needs propagation, i decided to rely on
         the forceCommandPropagation, and just fix the code to propagate MULTI when
         needed rather than depending on the command flags at all.
      
         The implication of my change then is that now it won't decide to propagate
         MULTI when it sees one of these: SELECT, PING, INFO, COMMAND, TIME and
         other commands which are neither read nor write.
      
      3. Changing getNodeByQuery and clusterRedirectBlockedClientIfNeeded in
         cluster.c to look at !write rather than read flag.
         This should have no implications, since these code paths are only reachable
         for commands which access keys, and these are always marked as either read
         or write.
      
      This commit improve MULTI propagation tests, for modules and a bunch of
      other special cases, all of which used to pass already before that commit.
      the only one that test change that uncovered a change of behavior is the
      one that DELs a non-existing key, it used to propagate an empty
      multi-exec block, and no longer does.
      411c18bb
  20. 14 Dec, 2020 1 commit
    • Oran Agra's avatar
      Tests: fix new defrag test to be skipped when not supported (#8185) · 7d9b09ad
      Oran Agra authored
      Additionally the older defrag tests are using an obsolete way to check
      if the defragger is suuported (the error no longer contains "DISABLED").
      this doesn't usually makes a difference since these tests are completely
      skipped if the allocator is not jemalloc, but that would fail if the
      allocator is a jemalloc that doesn't support defrag.
      7d9b09ad
  21. 13 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      Modules: add defrag API support. (#8149) · 63c1303c
      Yossi Gottlieb authored
      Add a new set of defrag functions that take a defrag context and allow
      defragmenting memory blocks and RedisModuleStrings.
      
      Modules can register a defrag callback which will be invoked when the
      defrag process handles globals.
      
      Modules with custom data types can also register a datatype-specific
      defrag callback which is invoked for keys that require defragmentation.
      The callback and associated functions support both one-step and
      multi-step options, depending on the complexity of the key as exposed by
      the free_effort callback.
      63c1303c
  22. 09 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      Add module data-type support for COPY. (#8112) · 4e064fba
      Yossi Gottlieb authored
      This adds a copy callback for module data types, in order to make
      modules compatible with the new COPY command.
      
      The callback is optional and COPY will fail for keys with data types
      that do not implement it.
      4e064fba
  23. 08 Dec, 2020 1 commit
    • Oran Agra's avatar
      Handle output buffer limits for Module blocked clients (#8141) · 48efc25f
      Oran Agra authored
      Module blocked clients cache the response in a temporary client,
      the reply list in this client would be affected by the recent fix
      in #7202, but when the reply is later copied into the real client,
      it would have bypassed all the checks for output buffer limit, which
      would have resulted in both: responding with a partial response to
      the client, and also not disconnecting it at all.
      48efc25f
  24. 06 Dec, 2020 1 commit
    • guybe7's avatar
      Make sure we do not propagate nested MULTI/EXEC (#8097) · 1df5bb56
      guybe7 authored
      One way this was happening is when a module issued an RM_Call which would inject MULTI.
      If the module command that does that was itself issued by something else that already did
      added MULTI (e.g. another module, or a Lua script), it would have caused nested MULTI.
      
      In fact the MULTI state in the client or the MULTI_EMITTED flag in the context isn't
      the right indication that we need to propagate MULTI or not, because on a nested calls
      (possibly a module action called by a keyspace event of another module action), these
      flags aren't retained / reflected.
      
      instead there's now a global propagate_in_transaction flag for that.
      
      in addition to that, we now have a global in_eval and in_exec flags, to serve the flags
      of RM_GetContextFlags, since their dependence on the current client is wrong for the same
      reasons mentioned above.
      1df5bb56
  25. 01 Dec, 2020 1 commit
    • Itamar Haber's avatar
      Adds pub/sub channel patterns to ACL (#7993) · c1b1e8c3
      Itamar Haber authored
      Fixes #7923.
      
      This PR appropriates the special `&` symbol (because `@` and `*` are taken),
      followed by a literal value or pattern for describing the Pub/Sub patterns that
      an ACL user can interact with. It is similar to the existing key patterns
      mechanism in function (additive) and implementation (copy-pasta). It also adds
      the allchannels and resetchannels ACL keywords, naturally.
      
      The default user is given allchannels permissions, whereas new users get
      whatever is defined by the acl-pubsub-default configuration directive. For
      backward compatibility in 6.2, the default of this directive is allchannels but
      this is likely to be changed to resetchannels in the next major version for
      stronger default security settings.
      
      Unless allchannels is set for the user, channel access permissions are checked
      as follows :
      * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the
        argumentative channel name(s) exists for the user.
      * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument
        literally exist(s) in the user's list.
      
      Such failures are logged to the ACL log.
      
      Runtime changes to channel permissions for a user with existing subscribing
      clients cause said clients to disconnect unless the new permissions permit the
      connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched
      literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be
      disconnected.
      
      Notes/questions:
      * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons
        for touching them.
      c1b1e8c3
  26. 17 Nov, 2020 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025) · d87a0d02
      Meir Shpilraien (Spielrein) authored
      
      
      Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
      the caller, who executes the command in this context, expects a reply.
      
      Today, LUA and MULTI have a special (and different) treatment to blocking commands:
      
      LUA   - Most commands are marked with no-script flag which are checked when executing
      and command from LUA, commands that are not marked (like XREAD) verify that their
      blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
      MULTI - Command that is going to block, first verify that the client is not inside
      multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
      return a result which is a match to the empty key with no timeout (for example blpop
      inside MULTI will act as lpop)
      For modules that perform RM_Call with blocking command, the returned results type is
      REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.
      
      Disadvantages of the current state are:
      
      No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
      Module can not safely execute blocking command (and get reply or error).
      Though It is true that modules are not like LUA or MULTI and should be smarter not
      to execute blocking commands on RM_Call, sometimes you want to execute a command base
      on client input (for example if you create a module that provides a new scripting
      language like javascript or python).
      While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
      REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
      check if the command came from another module using RM_Call. So there is no way
      for a module to know not to block another module RM_Call execution.
      
      This commit adds a way to unify the treatment for blocking clients by introducing
      a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
      turned on to signify that the client should not be blocked. A blocking command
      verifies that the flag is turned off before blocking. If a blocking command sees
      that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
      which are matches to empty key with no timeout (as MULTI does today).
      
      The new flag is checked on the following commands:
      
      List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
      Zset blocking commands: BZPOPMIN, BZPOPMAX
      Stream blocking commands: XREAD, XREADGROUP
      SUBSCRIBE, PSUBSCRIBE, MONITOR
      In addition, the new flag is turned on inside the AOF client, we do not want to
      block the AOF client to prevent deadlocks and commands ordering issues (and there
      is also an existing assert in the code that verifies it).
      
      To keep backward compatibility on LUA, all the no-script flags on existing commands
      were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.
      
      To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
      We added a special treatment on those commands to allow executing them on MULTI.
      
      The only backward compatibility issue that this PR introduces is that now MONITOR
      is not allowed inside MULTI.
      
      Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
      or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      d87a0d02
  27. 16 Nov, 2020 1 commit
    • chenyangyang's avatar
      Modules callbacks for lazy free effort, and unlink (#7912) · c1aaad06
      chenyangyang authored
      Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
      and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
      Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
      asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
      may soon be freed by a background thread.
      
      Add `lazyfreed_objects` info field, which represents the number of objects that have been
      lazyfreed since redis was started.
      
      Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
      `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
      which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
      c1aaad06
  28. 11 Nov, 2020 1 commit
  29. 11 Oct, 2020 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · adc3183c
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      adc3183c
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 0aec98dc
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      0aec98dc
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 907da058
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      907da058
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 7d117d75
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      7d117d75
  30. 20 Sep, 2020 1 commit
  31. 09 Sep, 2020 1 commit
  32. 06 Sep, 2020 1 commit
  33. 17 Aug, 2020 1 commit
  34. 09 Aug, 2020 1 commit
  35. 06 Aug, 2020 1 commit
    • Oran Agra's avatar
      Accelerate diskless master connections, and general re-connections (#6271) · c17e597d
      Oran Agra authored
      Diskless master has some inherent latencies.
      1) fork starts with delay from cron rather than immediately
      2) replica is put online only after an ACK. but the ACK
         was sent only once a second.
      3) but even if it would arrive immediately, it will not
         register in case cron didn't yet detect that the fork is done.
      
      Besides that, when a replica disconnects, it doesn't immediately
      attempts to re-connect, it waits for replication cron (one per second).
      in case it was already online, it may be important to try to re-connect
      as soon as possible, so that the backlog at the master doesn't vanish.
      
      In case it disconnected during rdb transfer, one can argue that it's
      not very important to re-connect immediately, but this is needed for the
      "diskless loading short read" test to be able to run 100 iterations in 5
      seconds, rather than 3 (waiting for replication cron re-connection)
      
      changes in this commit:
      1) sync command starts a fork immediately if no sync_delay is configured
      2) replica sends REPLCONF ACK when done reading the rdb (rather than on 1s cron)
      3) when a replica unexpectedly disconnets, it immediately tries to
         re-connect rather than waiting 1s
      4) when when a child exits, if there is another replica waiting, we spawn a new
         one right away, instead of waiting for 1s replicationCron.
      5) added a call to connectWithMaster from replicationSetMaster. which is called
         from the REPLICAOF command but also in 3 places in cluster.c, in all of
         these the connection attempt will now be immediate instead of delayed by 1
         second.
      
      side note:
      we can add a call to rdbPipeReadHandler in replconfCommand when getting
      a REPLCONF ACK from the replica to solve a race where the replica got
      the entire rdb and EOF marker before we detected that the pipe was
      closed.
      in the test i did see this race happens in one about of some 300 runs,
      but i concluded that this race is unlikely in real life (where the
      replica is on another host and we're more likely to first detect the
      pipe was closed.
      the test runs 100 iterations in 3 seconds, so in some cases it'll take 4
      seconds instead (waiting for another REPLCONF ACK).
      
      Removing unneeded startBgsaveForReplication from updateSlavesWaitingForBgsave
      Now that CheckChildrenDone is calling the new replicationStartPendingFork
      (extracted from serverCron) there's actually no need to call
      startBgsaveForReplication from updateSlavesWaitingForBgsave anymore,
      since as soon as updateSlavesWaitingForBgsave returns, CheckChildrenDone is
      calling replicationStartPendingFork that handles that anyway.
      The code in updateSlavesWaitingForBgsave had a bug in which it ignored
      repl-diskless-sync-delay, but removing that code shows that this bug was
      hiding another bug, which is that the max_idle should have used >= and
      not >, this one second delay has a big impact on my new test.
      c17e597d