1. 30 Mar, 2022 1 commit
    • Nick Chun's avatar
      Module Configurations (#10285) · bda9d74d
      Nick Chun authored
      
      
      This feature adds the ability to add four different types (Bool, Numeric,
      String, Enum) of configurations to a module to be accessed via the redis
      config file, and the CONFIG command.
      
      **Configuration Names**:
      
      We impose a restriction that a module configuration always starts with the
      module name and contains a '.' followed by the config name. If a module passes
      "config1" as the name to a register function, it will be registered as MODULENAME.config1.
      
      **Configuration Persistence**:
      
      Module Configurations exist only as long as a module is loaded. If a module is
      unloaded, the configurations are removed.
      There is now also a minimal core API for removal of standardConfig objects
      from configs by name.
      
      **Get and Set Callbacks**:
      
      Storage of config values is owned by the module that registers them, and provides
      callbacks for Redis to access and manipulate the values.
      This is exposed through a GET and SET callback.
      
      The get callback returns a typed value of the config to redis. The callback takes
      the name of the configuration, and also a privdata pointer. Note that these only
      take the CONFIGNAME portion of the config, not the entire MODULENAME.CONFIGNAME.
      
      ```
       typedef RedisModuleString * (*RedisModuleConfigGetStringFunc)(const char *name, void *privdata);
       typedef long long (*RedisModuleConfigGetNumericFunc)(const char *name, void *privdata);
       typedef int (*RedisModuleConfigGetBoolFunc)(const char *name, void *privdata);
       typedef int (*RedisModuleConfigGetEnumFunc)(const char *name, void *privdata);
      ```
      
      Configs must also must specify a set callback, i.e. what to do on a CONFIG SET XYZ 123
      or when loading configurations from cli/.conf file matching these typedefs. *name* is
      again just the CONFIGNAME portion, *val* is the parsed value from the core,
      *privdata* is the registration time privdata pointer, and *err* is for providing errors to a client.
      
      ```
      typedef int (*RedisModuleConfigSetStringFunc)(const char *name, RedisModuleString *val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetNumericFunc)(const char *name, long long val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetBoolFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetEnumFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
      ```
      
      Modules can also specify an optional apply callback that will be called after
      value(s) have been set via CONFIG SET:
      
      ```
      typedef int (*RedisModuleConfigApplyFunc)(RedisModuleCtx *ctx, void *privdata, RedisModuleString **err);
      ```
      
      **Flags:**
      We expose 7 new flags to the module, which are used as part of the config registration.
      
      ```
      #define REDISMODULE_CONFIG_MODIFIABLE 0 /* This is the default for a module config. */
      #define REDISMODULE_CONFIG_IMMUTABLE (1ULL<<0) /* Can this value only be set at startup? */
      #define REDISMODULE_CONFIG_SENSITIVE (1ULL<<1) /* Does this value contain sensitive information */
      #define REDISMODULE_CONFIG_HIDDEN (1ULL<<4) /* This config is hidden in `config get <pattern>` (used for tests/debugging) */
      #define REDISMODULE_CONFIG_PROTECTED (1ULL<<5) /* Becomes immutable if enable-protected-configs is enabled. */
      #define REDISMODULE_CONFIG_DENY_LOADING (1ULL<<6) /* This config is forbidden during loading. */
      /* Numeric Specific Configs */
      #define REDISMODULE_CONFIG_MEMORY (1ULL<<7) /* Indicates if this value can be set as a memory value */
      ```
      
      **Module Registration APIs**:
      
      ```
      int (*RedisModule_RegisterBoolConfig)(RedisModuleCtx *ctx, char *name, int default_val, unsigned int flags, RedisModuleConfigGetBoolFunc getfn, RedisModuleConfigSetBoolFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterNumericConfig)(RedisModuleCtx *ctx, const char *name, long long default_val, unsigned int flags, long long min, long long max, RedisModuleConfigGetNumericFunc getfn, RedisModuleConfigSetNumericFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterStringConfig)(RedisModuleCtx *ctx, const char *name, const char *default_val, unsigned int flags, RedisModuleConfigGetStringFunc getfn, RedisModuleConfigSetStringFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterEnumConfig)(RedisModuleCtx *ctx, const char *name, int default_val, unsigned int flags, const char **enum_values, const int *int_values, int num_enum_vals, RedisModuleConfigGetEnumFunc getfn, RedisModuleConfigSetEnumFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_LoadConfigs)(RedisModuleCtx *ctx);
      ```
      
      The module name will be auto appended along with a "." to the front of the name of the config.
      
      **What RM_Register[...]Config does**:
      
      A RedisModule struct now keeps a list of ModuleConfig objects which look like:
      ```
      typedef struct ModuleConfig {
          sds name; /* Name of config without the module name appended to the front */
          void *privdata; /* Optional data passed into the module config callbacks */
          union get_fn { /* The get callback specificed by the module */
              RedisModuleConfigGetStringFunc get_string;
              RedisModuleConfigGetNumericFunc get_numeric;
              RedisModuleConfigGetBoolFunc get_bool;
              RedisModuleConfigGetEnumFunc get_enum;
          } get_fn;
          union set_fn { /* The set callback specified by the module */
              RedisModuleConfigSetStringFunc set_string;
              RedisModuleConfigSetNumericFunc set_numeric;
              RedisModuleConfigSetBoolFunc set_bool;
              RedisModuleConfigSetEnumFunc set_enum;
          } set_fn;
          RedisModuleConfigApplyFunc apply_fn;
          RedisModule *module;
      } ModuleConfig;
      ```
      It also registers a standardConfig in the configs array, with a pointer to the
      ModuleConfig object associated with it.
      
      **What happens on a CONFIG GET/SET MODULENAME.MODULECONFIG:**
      
      For CONFIG SET, we do the same parsing as is done in config.c and pass that
      as the argument to the module set callback. For CONFIG GET, we call the
      module get callback and return that value to config.c to return to a client.
      
      **CONFIG REWRITE**:
      
      Starting up a server with module configurations in a .conf file but no module load
      directive will fail. The flip side is also true, specifying a module load and a bunch
      of module configurations will load those configurations in using the module defined
      set callbacks on a RM_LoadConfigs call. Configs being rewritten works the same
      way as it does for standard configs, as the module has the ability to specify a
      default value. If a module is unloaded with configurations specified in the .conf file
      those configurations will be commented out from the .conf file on the next config rewrite.
      
      **RM_LoadConfigs:**
      
      `RedisModule_LoadConfigs(RedisModuleCtx *ctx);`
      
      This last API is used to make configs available within the onLoad() after they have
      been registered. The expected usage is that a module will register all of its configs,
      then call LoadConfigs to trigger all of the set callbacks, and then can error out if any
      of them were malformed. LoadConfigs will attempt to set all configs registered to
      either a .conf file argument/loadex argument or their default value if an argument is
      not specified. **LoadConfigs is a required function if configs are registered.
      ** Also note that LoadConfigs **does not** call the apply callbacks, but a module
      can do that directly after the LoadConfigs call.
      
      **New Command: MODULE LOADEX [CONFIG NAME VALUE] [ARGS ...]:**
      
      This command provides the ability to provide startup context information to a module.
      LOADEX stands for "load extended" similar to GETEX. Note that provided config
      names need the full MODULENAME.MODULECONFIG name. Any additional
      arguments a module might want are intended to be specified after ARGS.
      Everything after ARGS is passed to onLoad as RedisModuleString **argv.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      Co-authored-by: default avatarMadelyn Olson <matolson@amazon.com>
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      bda9d74d
  2. 22 Mar, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Add new RM_Call flags for script mode, no writes, and error replies. (#10372) · f3855a09
      Meir Shpilraien (Spielrein) authored
      The PR extends RM_Call with 3 new capabilities using new flags that
      are given to RM_Call as part of the `fmt` argument.
      It aims to assist modules that are getting a list of commands to be
      executed from the user (not hard coded as part of the module logic),
      think of a module that implements a new scripting language...
      
      * `S` - Run the command in a script mode, this means that it will raise an
        error if a command which are not allowed inside a script (flaged with the
        `deny-script` flag) is invoked (like SHUTDOWN). In addition, on script mode,
        write commands are not allowed if there is not enough good replicas (as
        configured with `min-replicas-to-write`) and/or a disk error happened.
      
      * `W` - no writes mode, Redis will reject any command that is marked with `write`
        flag. Again can be useful to modules that implement a new scripting language
        and wants to prevent any write commands.
      
      * `E` - Return errors as RedisModuleCallReply. Today the errors that happened
        before the command was invoked (like unknown commands or acl error) return
        a NULL reply and set errno. This might be missing important information about
        the failure and it is also impossible to just pass the error to the user using
        RM_ReplyWithCallReply. This new flag allows you to get a RedisModuleCallReply
        object with the relevant error message and treat it as if it was an error that was
        raised by the command invocation.
      
      Tests were added to verify the new code paths.
      
      In addition small refactoring was done to share some code between modules,
      scripts, and `processCommand` function:
      1. `getAclErrorMessage` was added to `acl.c` to unified to log message extraction
        from the acl result
      2. `checkGoodReplicasStatus` was added to `replication.c` to check the status of
        good replicas. It is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and
        `processCommand`.
      3. `writeCommandsGetDiskErrorMessage` was added to `server.c` to get the error
        message on persistence failure. Again it is used on `scriptVerifyWriteCommandAllow`,
        `RM_Call`, and `processCommand`.
      f3855a09
  3. 18 Mar, 2022 1 commit
    • sundb's avatar
      Restore ::singledb after cluster test (#10441) · b9656adb
      sundb authored
      When ::singledb is 0, we will use db 9 for the test db.
      Since ::singledb is set to 1 in the cluster-related tests, but not restored, some subsequent
      tests associated with db 9 will fail.
      b9656adb
  4. 16 Mar, 2022 2 commits
    • Binbin's avatar
      Fix module redact test for valgrind (#10432) · 61b7e591
      Binbin authored
      The new module redact test will fail with valgrind:
      ```
      [err]: modules can redact arguments in tests/unit/moduleapi/auth.tcl
      Expected 'slowlog reset' to be equal to 'auth.redact 1 (redacted) 3 (redacted)' (context: type eval line 12 cmd {assert_equal {slowlog reset} [lindex [lindex [r slowlog get] 2] 3]} proc ::test)
      ```
      
      The reason is that with `slowlog-log-slower-than 10000`,
      `slowlog get` will have a chance to exceed 10ms.
      
      Made two changes to avoid failure:
      1. change `slowlog-log-slower-than` from 10000 to -1, distable it.
      2. assert to use the previous execution result.
      
      In theory, the second one can actually be left unchanged, but i
      think it will be better if it is changed.
      61b7e591
    • Madelyn Olson's avatar
      Add module API for redacting command arguments (#10425) · 416c9ac2
      Madelyn Olson authored
      Add module API for redacting client commands
      416c9ac2
  5. 08 Mar, 2022 1 commit
    • guybe7's avatar
      XREADGROUP: Unblock client if stream is deleted (#10306) · 2a295408
      guybe7 authored
      Deleting a stream while a client is blocked XREADGROUP should unblock the client.
      
      The idea is that if a client is blocked via XREADGROUP is different from
      any other blocking type in the sense that it depends on the existence of both
      the key and the group. Even if the key is deleted and then revived with XADD
      it won't help any clients blocked on XREADGROUP because the group no longer
      exist, so they would fail with -NOGROUP anyway.
      The conclusion is that it's better to unblock these clients (with error) upon
      the deletion of the key, rather than waiting for the first XADD. 
      
      Other changes:
      1. Slightly optimize all `serveClientsBlockedOn*` functions by checking `server.blocked_clients_by_type`
      2. All `serveClientsBlockedOn*` functions now use a list iterator rather than looking at `listFirst`, relying
        on `unblockClient` to delete the head of the list. Before this commit, only `serveClientsBlockedOnStreams`
        used to work like that.
      3. bugfix: CLIENT UNBLOCK ERROR should work even if the command doesn't have a timeout_callback
        (only relevant to module commands)
      2a295408
  6. 07 Mar, 2022 1 commit
  7. 22 Feb, 2022 1 commit
    • Madelyn Olson's avatar
      Implemented module getchannels api and renamed channel keyspec (#10299) · 71204f96
      Madelyn Olson authored
      This implements the following main pieces of functionality:
      * Renames key spec "CHANNEL" to be "NOT_KEY", and update the documentation to
        indicate it's for cluster routing and not for any other key related purpose.
      * Add the getchannels-api, so that modules can now define commands that are subject to
        ACL channel permission checks. 
      * Add 4 new flags that describe how a module interacts with a command (SUBSCRIBE, PUBLISH,
        UNSUBSCRIBE, and PATTERN). They are all technically composable, however not sure how a
        command could both subscribe and unsubscribe from a command at once, but didn't see
        a reason to add explicit validation there.
      * Add two new module apis RM_ChannelAtPosWithFlags and RM_IsChannelsPositionRequest to
        duplicate the functionality provided by the keys position APIs.
      * The RM_ACLCheckChannelPermissions (only released in 7.0 RC1) was changed to take flags
        rather than a boolean literal.
      * The RM_ACLCheckKeyPermissions (only released in 7.0 RC1) was changed to take flags
        corresponding to keyspecs instead of custom permission flags. These keyspec flags mimic
        the flags for ACLCheckChannelPermissions.
      71204f96
  8. 21 Feb, 2022 1 commit
    • Oran Agra's avatar
      Fix error stats and failed command stats for blocked clients (#10309) · fad0b0d2
      Oran Agra authored
      This is a followup work for #10278, and a discussion about #10279
      
      The changes:
      - fix failed_calls in command stats for blocked clients that got error.
        including CLIENT UNBLOCK, and module replying an error from a thread.
      - fix latency stats for XREADGROUP that filed with -NOGROUP
      
      Theory behind which errors should be counted:
      - error stats represents errors returned to the user, so an error handled by a
        module should not be counted.
      - total error counter should be the same.
      - command stats represents execution of commands (even with RM_Call, and if
        they fail or get rejected it counts these calls in commandstats, so it should
        also count failed_calls)
      
      Some thoughts about Scripts:
      for scripts it could be different since they're part of user code, not the infra (not an extension to redis)
      we certainly want commandstats to contain all calls and errors
      a simple script is like mult-exec transaction so an error inside it should be counted in error stats
      a script that replies with an error to the user (using redis.error_reply) should also be counted in error stats
      but then the problem is that a plain `return redis.call("SET")` should not be counted twice (once for the SET
      and once for EVAL)
      so that's something left to be resolved in #10279
      fad0b0d2
  9. 13 Feb, 2022 1 commit
    • Oran Agra's avatar
      Fix and improve module error reply statistics (#10278) · b099889a
      Oran Agra authored
      This PR handles several aspects
      1. Calls to RM_ReplyWithError from thread safe contexts don't violate thread safety.
      2. Errors returning from RM_Call to the module aren't counted in the statistics (they
        might be handled silently by the module)
      3. When a module propagates a reply it got from RM_Call to it's client, then the error
        statistics are counted.
      
      This is done by:
      1. When appending an error reply to the output buffer, we avoid updating the global
        error statistics, instead we cache that error in a deferred list in the client struct.
      2. When creating a RedisModuleCallReply object, the deferred error list is moved from
        the client into that object.
      3. when a module calls RM_ReplyWithCallReply we copy the deferred replies to the dest
        client (if that's a real client, then that's when the error statistics are updated to the server)
      
      Note about RM_ReplyWithCallReply: if the original reply had an array with errors, and the module
      replied with just a portion of the original reply, and not the entire reply, the errors are currently not
      propagated and the errors stats will not get propagated.
      
      Fix #10180
      b099889a
  10. 11 Feb, 2022 1 commit
    • sundb's avatar
      Fix duplicate module options define (#10284) · 5f0119ca
      sundb authored
      
      
      The bug is introduced by #9323. (released in 7.0 RC1)
      The define of `REDISMODULE_OPTIONS_HANDLE_IO_ERRORS` and `REDISMODULE_OPTION_NO_IMPLICIT_SIGNAL_MODIFIED` have the same value.
      
      This will result in skipping `signalModifiedKey()` after `RM_CloseKey()` if the module has set
      `REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD` option.
      The implication is missing WATCH and client side tracking invalidations.
      
      Other changes:
      - add `no-implicit-signal-modified` to the options in INFO modules
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5f0119ca
  11. 08 Feb, 2022 2 commits
    • Wen Hui's avatar
      Make INFO command variadic (#6891) · 2e1bc942
      Wen Hui authored
      
      
      This is an enhancement for INFO command, previously INFO only support one argument
      for different info section , if user want to get more categories information, either perform
      INFO all / default or calling INFO for multiple times.
      
      **Description of the feature**
      
      The goal of adding this feature is to let the user retrieve multiple categories via the INFO
      command, and still avoid emitting the same section twice.
      
      A use case for this is like Redis Sentinel, which periodically calling INFO command to refresh
      info from monitored Master/Slaves, only Server and Replication part categories are used for
      parsing information. If the INFO command can return just enough categories that client side
      needs, it can save a lot of time for client side parsing it as well as network bandwidth.
      
      **Implementation**
      To share code between redis, sentinel, and other users of INFO (DEBUG and modules),
      we have a new `genInfoSectionDict` function that returns a dict and some boolean flags
      (e.g. `all`) to the caller (built from user input).
      Sentinel is later purging unwanted sections from that, and then it is forwarded to the info `genRedisInfoString`.
      
      **Usage Examples**
      INFO Server Replication   
      INFO CPU Memory
      INFO default commandstats
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2e1bc942
    • Oran Agra's avatar
      Handle key-spec flags with modules (#10237) · 66be30f7
      Oran Agra authored
      - add COMMAND GETKEYSANDFLAGS sub-command
      - add RM_KeyAtPosWithFlags and GetCommandKeysWithFlags
      - RM_KeyAtPos and RM_CreateCommand set flags requiring full access for keys
      - RM_CreateCommand set VARIABLE_FLAGS
      - expose `variable_flags` flag in COMMAND INFO key-specs
      - getKeysFromCommandWithSpecs prefers key-specs over getkeys-api
      - add tests for all of these
      66be30f7
  12. 07 Feb, 2022 1 commit
    • Binbin's avatar
      COMMAND DOCS avoid adding summary/since if they don't exist (#10252) · 7f4cca11
      Binbin authored
      If summary or since is empty, we used to return NULL in
      COMMAND DOCS. Currently all redis commands will have these
      two fields.
      
      But not for module command, summary and since are optional
      for RM_SetCommandInfo. With the change in #10043, if a module
      command doesn't have the summary or since, redis-cli will
      crash (see #10250).
      
      In this commit, COMMAND DOCS avoid adding summary or since
      when they are missing.
      7f4cca11
  13. 04 Feb, 2022 1 commit
    • Viktor Söderqvist's avatar
      Command info module API (#10108) · 0a82fe84
      Viktor Söderqvist authored
      Adds RM_SetCommandInfo, allowing modules to provide the following command info:
      
      * summary
      * complexity
      * since
      * history
      * hints
      * arity
      * key specs
      * args
      
      This information affects the output of `COMMAND`, `COMMAND INFO` and `COMMAND DOCS`,
      Cluster, ACL and is used to filter commands with the wrong number of arguments before
      the call reaches the module code.
      
      The recently added API functions for key specs (never released) are removed.
      
      A minimalist example would look like so:
      ```c
          RedisModuleCommand *mycmd = RedisModule_GetCommand(ctx,"mymodule.mycommand");
          RedisModuleCommandInfo mycmd_info = {
              .version = REDISMODULE_COMMAND_INFO_VERSION,
              .arity = -5,
              .summary = "some description",
          };
          if (RedisModule_SetCommandInfo(mycmd, &mycmd_info) == REDISMODULE_ERR)
              return REDISMODULE_ERR;
      ````
      
      Notes:
      * All the provided information (including strings) is copied, not keeping references to the API input data.
      * The version field is actually a static struct that contains the sizes of the the structs used in arrays,
        so we can extend these in the future and old version will still be able to take the part they can support.
      0a82fe84
  14. 01 Feb, 2022 1 commit
  15. 26 Jan, 2022 1 commit
    • Binbin's avatar
      Allow SET without GET arg on write-only ACL. Allow BITFIELD GET on read-only ACL (#10148) · d6169258
      Binbin authored
      
      
      SET is a R+W command, because it can also do `GET` on the data.
      SET without GET is a write-only command.
      SET with GET is a read+write command.
      
      In #9974, we added ACL to let users define write-only access.
      So when the user uses SET with GET option, and the user doesn't
      have the READ permission on the key, we need to reject it,
      but we rather not reject users with write-only permissions from using
      the SET command when they don't use GET.
      
      In this commit, we add a `getkeys_proc` function to control key
      flags in SET command. We also add a new key spec flag (VARIABLE_FLAGS)
      means that some keys might have different flags depending on arguments.
      
      We also handle BITFIELD command, add a `bitfieldGetKeys` function.
      BITFIELD GET is a READ ONLY command.
      BITFIELD SET or BITFIELD INCR are READ WRITE commands.
      
      Other changes:
      1. SET GET was added in 6.2, add the missing since in set.json
      2. Added tests to cover the changes in acl-v2.tcl
      3. Fix some typos in server.h and cleanups in acl-v2.tcl
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      d6169258
  16. 24 Jan, 2022 1 commit
  17. 23 Jan, 2022 1 commit
    • Binbin's avatar
      sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) · 23325c13
      Binbin authored
      
      
      Summary of changes:
      1. Rename `redisCommand->name` to `redisCommand->declared_name`, it is a
        const char * for native commands and SDS for module commands.
      2. Store the [sub]command fullname in `redisCommand->fullname` (sds).
      3. List subcommands in `ACL CAT`
      4. List subcommands in `COMMAND LIST`
      5. `moduleUnregisterCommands` now will also free the module subcommands.
      6. RM_GetCurrentCommandName returns full command name
      
      Other changes:
      1. Add `addReplyErrorArity` and `addReplyErrorExpireTime`
      2. Remove `getFullCommandName` function that now is useless.
      3. Some cleanups about `fullname` since now it is SDS.
      4. Delete `populateSingleCommand` function from server.h that is useless.
      5. Added tests to cover this change.
      6. Add some module unload tests and fix the leaks
      7. Make error messages uniform, make sure they always contain the full command
        name and that it's quoted.
      7. Fixes some typos
      
      see the history in #9504, fixes #10124
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarguybe7 <guy.benoish@redislabs.com>
      23325c13
  18. 20 Jan, 2022 2 commits
    • Madelyn Olson's avatar
      ACL V2 - Selectors and key based permissions (#9974) · 55c81f2c
      Madelyn Olson authored
      
      
      * Implemented selectors which provide multiple different sets of permissions to users
      * Implemented key based permissions 
      * Added a new ACL dry-run command to test permissions before execution
      * Updated module APIs to support checking key based permissions
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      55c81f2c
    • perryitay's avatar
      Adding module api for processing commands during busy jobs and allow flagging... · c4b78823
      perryitay authored
      
      Adding module api for processing commands during busy jobs and allow flagging the commands that should be handled at this status (#9963)
      
      Some modules might perform a long-running logic in different stages of Redis lifetime, for example:
      * command execution
      * RDB loading
      * thread safe context
      
      During this long-running logic Redis is not responsive.
      
      This PR offers 
      1. An API to process events while a busy command is running (`RM_Yield`)
      2. A new flag (`ALLOW_BUSY`) to mark the commands that should be handled during busy
        jobs which can also be used by modules (`allow-busy`)
      3. In slow commands and thread safe contexts, this flag will start rejecting commands with -BUSY only
        after `busy-reply-threshold`
      4. During loading (`rdb_load` callback), it'll process events right away (not wait for `busy-reply-threshold`),
        but either way, the processing is throttled to the server hz rate.
      5. Allow modules to Yield to redis background tasks, but not to client commands
      
      * rename `script-time-limit` to `busy-reply-threshold` (an alias to the pre-7.0 `lua-time-limit`)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      c4b78823
  19. 19 Jan, 2022 1 commit
  20. 18 Jan, 2022 2 commits
    • Oran Agra's avatar
      New detailed key-spec flags (RO, RW, OW, RM, ACCESS, UPDATE, INSERT, DELETE) (#10122) · eef9c6b0
      Oran Agra authored
      The new ACL key based permissions in #9974 require the key-specs (#8324) to have more
      explicit flags rather than just READ and WRITE. See discussion in #10040
      
      This PR defines two groups of flags:
      One about how redis internally handles the key (mutually-exclusive).
      The other is about the logical operation done from the user's point of view (3 mutually exclusive
      write flags, and one read flag, all optional).
      In both groups, if we can't explicitly flag something as explicit read-only, delete-only, or
      insert-only, we flag it as `RW` or `UPDATE`.
      here's the definition from the code:
      ```
      /* Key-spec flags *
       * -------------- */
      /* The following refer what the command actually does with the value or metadata
       * of the key, and not necessarily the user data or how it affects it.
       * Each key-spec may must have exaclty one of these. Any operation that's not
       * distinctly deletion, overwrite or read-only would be marked as RW. */
      #define CMD_KEY_RO (1ULL<<0)     /* Read-Only - Reads the value of the key, but
                                        * doesn't necessarily returns it. */
      #define CMD_KEY_RW (1ULL<<1)     /* Read-Write - Modifies the data stored in the
                                        * value of the key or its metadata. */
      #define CMD_KEY_OW (1ULL<<2)     /* Overwrite - Overwrites the data stored in
                                        * the value of the key. */
      #define CMD_KEY_RM (1ULL<<3)     /* Deletes the key. */
      /* The follwing refer to user data inside the value of the key, not the metadata
       * like LRU, type, cardinality. It refers to the logical operation on the user's
       * data (actual input strings / TTL), being used / returned / copied / changed,
       * It doesn't refer to modification or returning of metadata (like type, count,
       * presence of data). Any write that's not INSERT or DELETE, would be an UPADTE.
       * Each key-spec may have one of the writes with or without access, or none: */
      #define CMD_KEY_ACCESS (1ULL<<4) /* Returns, copies or uses the user data from
                                        * the value of the key. */
      #define CMD_KEY_UPDATE (1ULL<<5) /* Updates data to the value, new value may
                                        * depend on the old value. */
      #define CMD_KEY_INSERT (1ULL<<6) /* Adds data to the value with no chance of,
                                        * modification or deletion of existing data. */
      #define CMD_KEY_DELETE (1ULL<<7) /* Explicitly deletes some content
                                        * from the value of the key. */
      ```
      
      Unrelated changes:
      - generate-command-code.py is only compatible with python3 (modified the shabang)
      - generate-command-code.py print file on json parsing error
      - rename `shard_channel` key-spec flag to just `channel`.
      - add INCOMPLETE flag in input spec of SORT and SORT_RO
      eef9c6b0
    • Ozan Tezcan's avatar
      Add event loop support to the module API (#10001) · 99ab4236
      Ozan Tezcan authored
      Modules can now register sockets/pipe to the Redis main thread event loop and do network operations asynchronously. Previously, modules had to maintain an event loop and another thread for asynchronous network operations.
      
      Also, if a module is calling API functions after doing some network operations, it had to synchronize its event loop thread's access with Redis main thread by locking the GIL, causing contention on the lock. After this commit, no synchronization is needed as module can operate in Redis main thread context. So, this commit may improve the performance for some use cases.
      
      Added three functions to the module API:
      
      * RedisModule_EventLoopAdd(int fd, int mask, RedisModuleEventLoopFunc func, void *user_data)
      * RedisModule_EventLoopDel(int fd, int mask)
      * RedisModule_EventLoopAddOneShot(RedisModuleEventLoopOneShotFunc func, void *user_data) - This function can be called from other threads to trigger callback on Redis main thread. Callback will be triggered only once. If Redis main thread is sleeping, this call will wake up the Redis main thread.
      Event loop callbacks are called by Redis main thread after locking the GIL. Inside callbacks, modules can operate as if they are holding the GIL.
      
      Added REDISMODULE_EVENT_EVENTLOOP event with two subevents:
      
      * REDISMODULE_SUBEVENT_EVENTLOOP_BEFORE_SLEEP
      * REDISMODULE_SUBEVENT_EVENTLOOP_AFTER_SLEEP
      
      These events are for modules that want to participate in the before and after sleep action. e.g It might be useful to implement batching : Read data from the network, write all to a file in one go on BEFORE_SLEEP event.
      99ab4236
  21. 17 Jan, 2022 1 commit
    • Oran Agra's avatar
      Set repl-diskless-sync to yes by default, add repl-diskless-sync-max-replicas (#10092) · ae899589
      Oran Agra authored
      1. enable diskless replication by default
      2. add a new config named repl-diskless-sync-max-replicas that enables
         replication to start before the full repl-diskless-sync-delay was
         reached.
      3. put replica online sooner on the master (see below)
      4. test suite uses repl-diskless-sync-delay of 0 to be faster
      5. a few tests that use multiple replica on a pre-populated master, are
         now using the new repl-diskless-sync-max-replicas
      6. fix possible timing issues in a few cluster tests (see below)
      
      put replica online sooner on the master 
      ----------------------------------------------------
      there were two tests that failed because they needed for the master to
      realize that the replica is online, but the test code was actually only
      waiting for the replica to realize it's online, and in diskless it could
      have been before the master realized it.
      
      changes include two things:
      1. the tests wait on the right thing
      2. issues in the master, putting the replica online in two steps.
      
      the master used to put the replica as online in 2 steps. the first
      step was to mark it as online, and the second step was to enable the
      write event (only after getting ACK), but in fact the first step didn't
      contains some of the tasks to put it online (like updating good slave
      count, and sending the module event). this meant that if a test was
      waiting to see that the replica is online form the point of view of the
      master, and then confirm that the module got an event, or that the
      master has enough good replicas, it could fail due to timing issues.
      
      so now the full effect of putting the replica online, happens at once,
      and only the part about enabling the writes is delayed till the ACK.
      
      fix cluster tests 
      --------------------
      I added some code to wait for the replica to sync and avoid race
      conditions.
      later realized the sentinel and cluster tests where using the original 5
      seconds delay, so changed it to 0.
      
      this means the other changes are probably not needed, but i suppose
      they're still better (avoid race conditions)
      ae899589
  22. 13 Jan, 2022 2 commits
    • Ozan Tezcan's avatar
      Added RM_MonotonicMicroseconds() API to provide monotonic time function (#10101) · f41cc870
      Ozan Tezcan authored
      Added RM_MonotonicMicroseconds(). Modules can use monotonic timestamp counter for measurements.
      f41cc870
    • chenyang8094's avatar
      Always create base AOF file when redis start from empty. (#10102) · e9bff797
      chenyang8094 authored
      
      
      Force create a BASE file (use a foreground `rewriteAppendOnlyFile`) when redis starts from an
      empty data set and  `appendonly` is  yes.
      
      The reasoning is that normally, after redis is running for some time, and the AOF has gone though
      a few rewrites, there's always a base rdb file. and the scenario where the base file is missing, is
      kinda rare (happens only at empty startup), so this change normalizes it.
      But more importantly, there are or could be some complex modules that are started with some
      configuration, when they create persistence they write that configuration to RDB AUX fields, so
      that can can always know with which configuration the persistence file they're loading was
      created (could be critical). there is (was) one scenario in which they could load their persisted data,
      and that configuration was missing, and this change fixes it.
      
      Add a new module event: REDISMODULE_SUBEVENT_PERSISTENCE_SYNC_AOF_START, similar to
      REDISMODULE_SUBEVENT_PERSISTENCE_AOF_START which is async.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      e9bff797
  23. 11 Jan, 2022 1 commit
    • Oran Agra's avatar
      Move doc metadata from COMMAND to COMMAND DOCS (#10056) · 3204a035
      Oran Agra authored
      Syntax:
      `COMMAND DOCS [<command name> ...]`
      
      Background:
      Apparently old version of hiredis (and thus also redis-cli) can't
      support more than 7 levels of multi-bulk nesting.
      
      The solution is to move all the doc related metadata from COMMAND to a
      new COMMAND DOCS sub-command.
      
      The new DOCS sub-command returns a map of commands (not an array like in COMMAND),
      And the same goes for the `subcommands` field inside it (also contains a map)
      
      Besides that, the remaining new fields of COMMAND (hints, key-specs, and
      sub-commands), are placed in the outer array rather than a nested map.
      this was done mainly for consistency with the old format.
      
      Other changes:
      ---
      * Allow COMMAND INFO with no arguments, which returns all commands, so that we can some day deprecated
        the plain COMMAND (no args)
      
      * Reduce the amount of deferred replies from both COMMAND and COMMAND
        DOCS, especially in the inner loops, since these create many small
        reply objects, which lead to many small write syscalls and many small
        TCP packets.
        To make this easier, when populating the command table, we count the
        history, args, and hints so we later know their size in advance.
        Additionally, the movablekeys flag was moved into the flags register.
      * Update generate-commands-json.py to take the data from both command, it
        now executes redis-cli directly, instead of taking input from stdin.
      * Sub-commands in both COMMAND (and COMMAND INFO), and also COMMAND DOCS,
        show their full name. i.e. CONFIG 
      *   GET will be shown as `config|get` rather than just `get`.
        This will be visible both when asking for `COMMAND INFO config` and COMMAND INFO config|get`, but is
        especially important for the later.
        i.e. imagine someone doing `COMMAND INFO slowlog|get config|get` not being able to distinguish between the two
        items in the array response.
      3204a035
  24. 10 Jan, 2022 1 commit
  25. 06 Jan, 2022 1 commit
    • Ozan Tezcan's avatar
      Set errno to EEXIST in redisFork() if child process exists (#10059) · 568c2e03
      Ozan Tezcan authored
      Callers of redisFork() are logging `strerror(errno)` on failure.
      `errno` is not set when there is already a child process, causing printing
      current value of errno which was set before `redisFork()` call. 
      
      Setting errno to EEXIST on this failure to provide more meaningful error message. 
      568c2e03
  26. 05 Jan, 2022 1 commit
    • filipe oliveira's avatar
      Added INFO LATENCYSTATS section: latency by percentile distribution/latency by... · 5dd15443
      filipe oliveira authored
      
      Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462)
      
      # Short description
      
      The Redis extended latency stats track per command latencies and enables:
      - exporting the per-command percentile distribution via the `INFO LATENCYSTATS` command.
        **( percentile distribution is not mergeable between cluster nodes ).**
      - exporting the per-command cumulative latency distributions via the `LATENCY HISTOGRAM` command.
        Using the cumulative distribution of latencies we can merge several stats from different cluster nodes
        to calculate aggregate metrics .
      
      By default, the extended latency monitoring is enabled since the overhead of keeping track of the
      command latency is very small.
       
      If you don't want to track extended latency metrics, you can easily disable it at runtime using the command:
       - `CONFIG SET latency-tracking no`
      
      By default, the exported latency percentiles are the p50, p99, and p999.
      You can alter them at runtime using the command:
      - `CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"`
      
      
      ## Some details:
      - The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command
        was called for the first time.
      - With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable
        ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable
        vs this branch. 
      - We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf )
      
      ## `INFO LATENCYSTATS` exposition format
      
         - Format: `latency_percentiles_usec_<CMDNAME>:p0=XX,p50....` 
      
      ## `LATENCY HISTOGRAM [command ...]` exposition format
      
      Return a cumulative distribution of latencies in the format of a histogram for the specified command names.
      
      The histogram is composed of a map of time buckets:
      - Each representing a latency range, between 1 nanosecond and roughly 1 second.
      - Each bucket covers twice the previous bucket's range.
      - Empty buckets are not printed.
      - Everything above 1 sec is considered +Inf.
      - At max there will be log2(1000000000)=30 buckets
      
      We reply a map for each command in the format:
      `<command name> : { `calls`: <total command calls> , `histogram` : { <bucket 1> : latency , < bucket 2> : latency, ...  } }`
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      5dd15443
  27. 04 Jan, 2022 1 commit
    • zhaozhao.zz's avatar
      use startEvictionTimeProc() in config set maxmemory (#10019) · 2e1979a2
      zhaozhao.zz authored
      This would mean that the effects of `CONFIG SET maxmemory` may not be visible once the command returns.
      That could anyway happen since incremental eviction was added in redis 6.2 (see #7653)
      
      We do this to fix one of the propagation bugs about eviction see #9890 and #10014.
      2e1979a2
  28. 28 Dec, 2021 1 commit
    • guybe7's avatar
      Remove incomplete fix of a broader problem (#10013) · 266d9506
      guybe7 authored
      Preventing COFIG SET maxmemory from propagating is just the tip of the iceberg.
      Module that performs a write operation in a notification can cause any
      command to be propagated, based on server.dirty
      
      We need to come up with a better solution.
      266d9506
  29. 27 Dec, 2021 3 commits
    • chenyang8094's avatar
      Tests: don't rely on the response of MEMORY USAGE when mem_allocator is not jemalloc (#10010) · af0b50f8
      chenyang8094 authored
      
      
      It turns out that libc malloc can return an allocation of a different size on requests of the same size.
      this means that matching MEMORY USAGE of one key to another copy of the same data can fail.
      
      Solution:
      Keep running the test that calls MEMORY USAGE, but ignore the response.
      We do that by introducing a new utility function to get the memory usage, which always returns 1
      when the allocator is not jemalloc.
      
      Other changes:
      Some formatting for datatype2.tcl
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      af0b50f8
    • Itamar Haber's avatar
      Adds utils/gen-commands-json.py (#9958) · f810510b
      Itamar Haber authored
      Following #9656, this script generates a "commands.json" file from the output
      of the new COMMAND. The output of this script is used in redis/redis-doc#1714
      and by redis/redis-io#259. This also converts a couple of rogue dashes (in 
      'key-specs' and 'multiple-token' flags) to underscores (continues #9959).
      f810510b
    • guybe7's avatar
      Fix race in propagation test (#10012) · 0f15e025
      guybe7 authored
      There's a race between testing DBSIZE and the thread starting.
      If the thread hadn't started by the time we checked DBISZE, no
      keys will have been evicted.
      The correct way is to check the evicted_keys stat.
      0f15e025
  30. 22 Dec, 2021 1 commit
    • guybe7's avatar
      Sort out mess around propagation and MULTI/EXEC (#9890) · 7ac21307
      guybe7 authored
      The mess:
      Some parts use alsoPropagate for late propagation, others using an immediate one (propagate()),
      causing edge cases, ugly/hacky code, and the tendency for bugs
      
      The basic idea is that all commands are propagated via alsoPropagate (i.e. added to a list) and the
      top-most call() is responsible for going over that list and actually propagating them (and wrapping
      them in MULTI/EXEC if there's more than one command). This is done in the new function,
      propagatePendingCommands.
      
      Callers to propagatePendingCommands:
      1. top-most call() (we want all nested call()s to add to the also_propagate array and just the top-most
         one to propagate them) - via `afterCommand`
      2. handleClientsBlockedOnKeys: it is out of call() context and it may propagate stuff - via `afterCommand`. 
      3. handleClientsBlockedOnKeys edge case: if the looked-up key is already expired, we will propagate the
         expire but will not unblock any client so `afterCommand` isn't called. in that case, we have to propagate
         the deletion explicitly.
      4. cron stuff: active-expire and eviction may also propagate stuff
      5. modules: the module API allows to propagate stuff from just about anywhere (timers, keyspace notifications,
         threads). I could have tried to catch all the out-of-call-context places but it seemed easier to handle it in one
         place: when we free the context. in the spirit of what was done in call(), only the top-most freeing of a module
         context may cause propagation.
      6. modules: when using a thread-safe ctx it's not clear when/if the ctx will be freed. we do know that the module
         must lock the GIL before calling RM_Replicate/RM_Call so we propagate the pending commands when
         releasing the GIL.
      
      A "known limitation", which were actually a bug, was fixed because of this commit (see propagate.tcl):
         When using a mix of RM_Call with `!` and RM_Replicate, the command would propagate out-of-order:
         first all the commands from RM_Call, and then the ones from RM_Replicate
      
      Another thing worth mentioning is that if, in the past, a client would issue a MULTI/EXEC with just one
      write command the server would blindly propagate the MULTI/EXEC too, even though it's redundant.
      not anymore.
      
      This commit renames propagate() to propagateNow() in order to cause conflicts in pending PRs.
      propagatePendingCommands is the only caller of propagateNow, which is now a static, internal helper function.
      
      Optimizations:
      1. alsoPropagate will not add stuff to also_propagate if there's no AOF and replicas
      2. alsoPropagate reallocs also_propagagte exponentially, to save calls to memmove
      
      Bugfixes:
      1. CONFIG SET can create evictions, sending notifications which can cause to dirty++ with modules.
         we need to prevent it from propagating to AOF/replicas
      2. We need to set current_client in RM_Call. buggy scenario:
         - CONFIG SET maxmemory, eviction notifications, module hook calls RM_Call
         - assertion in lookupKey crashes, because current_client has CONFIG SET, which isn't CMD_WRITE
      3. minor: in eviction, call propagateDeletion after notification, like active-expire and all commands
         (we always send a notification before propagating the command)
      7ac21307
  31. 19 Dec, 2021 2 commits
    • Oran Agra's avatar
      Add external test that runs without debug command (#9964) · 6add1b72
      Oran Agra authored
      - add needs:debug flag for some tests
      - disable "save" in external tests (speedup?)
      - use debug_digest proc instead of debug command directly so it can be skipped
      - use OBJECT ENCODING instead of DEBUG OBJECT to get encoding
      - add a proc for OBJECT REFCOUNT so it can be skipped
      - move a bunch of tests in latency_monitor tests to happen later so that latency monitor has some values in it
      - add missing close_replication_stream calls
      - make sure to close the temp client if DEBUG LOG fails
      6add1b72
    • YaacovHazan's avatar
      Protected configs and sensitive commands (#9920) · ae2f5b7b
      YaacovHazan authored
      Block sensitive configs and commands by default.
      
      * `enable-protected-configs` - block modification of configs with the new `PROTECTED_CONFIG` flag.
         Currently we add this flag to `dbfilename`, and `dir` configs,
         all of which are non-mutable configs that can set a file redis will write to.
      * `enable-debug-command` - block the `DEBUG` command
      * `enable-module-command` - block the `MODULE` command
      
      These have a default value set to `no`, so that these features are not
      exposed by default to client connections, and can only be set by modifying the config file.
      
      Users can change each of these to either `yes` (allow all access), or `local` (allow access from
      local TCP connections and unix domain connections)
      
      Note that this is a **breaking change** (specifically the part about MODULE command being disabled by default).
      I.e. we don't consider DEBUG command being blocked as an issue (people shouldn't have been using it),
      and the few configs we protected are unlikely to have been set at runtime anyway.
      On the other hand, it's likely to assume some users who use modules, load them from the config file anyway.
      Note that's the whole point of this PR, for redis to be more secure by default and reduce the attack surface on
      innocent users, so secure defaults will necessarily mean a breaking change.
      ae2f5b7b
  32. 18 Dec, 2021 1 commit