1. 26 Apr, 2022 1 commit
    • Madelyn Olson's avatar
      By default prevent cross slot operations in functions and scripts with # (#10615) · efcd1bf3
      Madelyn Olson authored
      Adds the `allow-cross-slot-keys` flag to Eval scripts and Functions to allow
      scripts to access keys from multiple slots.
      The default behavior is now that they are not allowed to do that (unlike before).
      This is a breaking change for 7.0 release candidates (to be part of 7.0.0), but
      not for previous redis releases since EVAL without shebang isn't doing this check.
      
      Note that the check is done on both the keys declared by the EVAL / FCALL command
      arguments, and also the ones used by the script when making a `redis.call`.
      
      A note about the implementation, there seems to have been some confusion
      about allowing access to non local keys. I thought I missed something in our
      wider conversation, but Redis scripts do block access to non-local keys.
      So the issue was just about cross slots being accessed.
      efcd1bf3
  2. 25 Apr, 2022 2 commits
    • guybe7's avatar
      Fix regression not aborting transaction on error, and re-edit some error responses (#10612) · df787764
      guybe7 authored
      1. Disk error and slave count checks didn't flag the transactions or counted correctly in command stats (regression from #10372  , 7.0 RC3)
      2. RM_Call will reply the same way Redis does, in case of non-exisitng command or arity error
      3. RM_WrongArtiy will consider the full command name
      4. Use lowercase 'u' in "unknonw subcommand" (to align with "unknown command")
      
      Followup work of #10127
      df787764
    • guybe7's avatar
      Test: RM_Call from within "expired" notification (#10613) · 21e39ec4
      guybe7 authored
      This case is interesting because it originates from cron,
      rather than from another command.
      
      The idea came from looking at #9890 and #10573, and I was wondering if RM_Call
      would work properly when `server.current_client == NULL`
      21e39ec4
  3. 24 Apr, 2022 2 commits
    • Yossi Gottlieb's avatar
      Run large-memory tests as solo. (#10626) · bd823c7f
      Yossi Gottlieb authored
      This avoids random memory spikes and enables --large-memory tests to run
      on moderately sized systems.
      bd823c7f
    • Binbin's avatar
      Fix timing issue in slowlog redact test (#10614) · a6b3ce28
      Binbin authored
      * Fix timing issue in slowlog redact test
      
      This test failed once in my daily CI (test-sanitizer-address (clang))
      ```
      *** [err]: SLOWLOG - Some commands can redact sensitive fields in tests/unit/slowlog.tcl
      Expected 'migrate 127.0.0.1 25649 key 9 5000 AUTH2 (redacted) (redacted)' to match '* key 9 5000 AUTH (redacted)' (context: type eval line 12 cmd {assert_match {* key 9 5000 AUTH (redacted)} [lindex [lindex [r slowlog get] 1] 3]} proc ::test)
      ```
      
      The reason is that with slowlog-log-slower-than 10000,
      slowlog get will have a chance to exceed 10ms.
      
      Change slowlog-log-slower-than from 10000 to -1, disable it.
      Also handles a same potentially problematic test above.
      This is actually the same timing issue as #10432.
      
      But also avoid repeated calls to `SLOWLOG GET`
      a6b3ce28
  4. 20 Apr, 2022 1 commit
  5. 19 Apr, 2022 2 commits
  6. 18 Apr, 2022 1 commit
    • Oran Agra's avatar
      Fix RM_Yield bug processing future commands of the current client. (#10573) · 7d1ad6ca
      Oran Agra authored
      RM_Yield was missing a call to protectClient to prevent redis from
      processing future commands of the yielding client.
      
      Adding tests that fail without this fix.
      
      This would be complicated to solve since nested calls to RM_Call used to
      replace the current_client variable with the module temp client.
      
      It looks like it's no longer necessary to do that, since it was added
      back in #9890 to solve two issues, both already gone:
      1. call to CONFIG SET maxmemory could trigger a module hook calling
         RM_Call. although this specific issue is gone, arguably other hooks
         like keyspace notification, can do the same.
      2. an assertion in lookupKey that checks the current command of the
         current client, introduced in #9572 and removed in #10248
      7d1ad6ca
  7. 17 Apr, 2022 3 commits
    • guybe7's avatar
      Add RM_PublishMessageShard (#10543) · f49ff156
      guybe7 authored
      since PUBLISH and SPUBLISH use different dictionaries for channels and clients,
      and we already have an API for PUBLISH, it only makes sense to have one for SPUBLISH
      
      Add test coverage and unifying some test infrastructure.
      f49ff156
    • Meir Shpilraien (Spielrein)'s avatar
      Added test to verify loading Lua binary payload is not possible (#10583) · 789c94fe
      Meir Shpilraien (Spielrein) authored
      The tests verify that loading a binary payload to the Lua interpreter raises an error.
      The Lua code modification was done here: fdf9d455
      which force the Lau interpreter to always use the text parser.
      789c94fe
    • guybe7's avatar
      Add RM_MallocSizeString, RM_MallocSizeDict (#10542) · fe1c096b
      guybe7 authored
      Add APIs to allow modules to compute the memory consumption of opaque objects owned by redis.
      Without these, the mem_usage callbacks of module data types are useless in many cases.
      
      Other changes:
      Fix streamRadixTreeMemoryUsage to include the size of the rax structure itself
      fe1c096b
  8. 14 Apr, 2022 2 commits
    • Madelyn Olson's avatar
      Fix incorrect error code for eval scripts and fix test error checking (#10575) · effa707e
      Madelyn Olson authored
      By the convention of errors, there is supposed to be a space between the code and the name.
      While looking at some lua stuff I noticed that interpreter errors were not adding the space,
      so some clients will try to map the detailed error message into the error.
      
      We have tests that hit this condition, but they were just checking that the string "starts" with ERR.
      I updated some other tests with similar incorrect string checking. This isn't complete though, as
      there are other ways we check for ERR I didn't fix.
      
      Produces some fun output like:
      ```
      # Errorstats
      errorstat_ERR:count=1
      errorstat_ERRuser_script_1_:count=1
      ```
      effa707e
    • Oran Agra's avatar
      solve corrupt dump fuzzer crash in streams (#10579) · 95050f26
      Oran Agra authored
      we had a panic in streamLastValidID when the stream metadata
      said it's not empty, but the rax is empty.
      95050f26
  9. 13 Apr, 2022 1 commit
    • Luke Palmer's avatar
      Keyspace event for new keys (#10512) · bb7891f0
      Luke Palmer authored
      Add an optional keyspace event when new keys are added to the db.
      
      This is useful for applications where clients need to be aware of the redis keyspace.
      Such an application can SCAN once at startup and then listen for "new" events (plus
      others associated with DEL, RENAME, etc).
      bb7891f0
  10. 12 Apr, 2022 1 commit
  11. 10 Apr, 2022 1 commit
    • guybe7's avatar
      COMMAND DOCS shows module name, where applicable (#10544) · 719db14e
      guybe7 authored
      Add field to COMMAND DOCS response to denote the name of the module
      that added that command.
      COMMAND LIST can filter by module, but if you get the full commands list,
      you may still wanna know which command belongs to which module.
      The alternative would be to do MODULE LIST, and then multiple calls to COMMAND LIST
      719db14e
  12. 07 Apr, 2022 1 commit
    • Oran Agra's avatar
      Fix RM_Yield bug (#10548) · 451531f1
      Oran Agra authored
      The bug was when using REDISMODULE_YIELD_FLAG_CLIENTS.
      in that case we would have only set the CLIENTS type flag in
      server.busy_module_yield_flags and then clear that flag when exiting
      RM_Yield, so we would never call unblockPostponedClients when the
      context is destroyed.
      
      This didn't really have any actual implication, which is why the tests
      couldn't (and still can't) find that since the bug only happens when
      using CLIENT, but in this case we won't have any clients to un-postpone
      i.e. clients will get rejected with BUSY error, rather than being
      postponed.
      
      Unrelated:
      * Adding tests for nested contexts, just in case.
      * Avoid nested RM_Yield calls
      451531f1
  13. 05 Apr, 2022 5 commits
    • Lu JJ's avatar
      Fix the bug that caused hash encoding errors when using hincrbyfloat or hincrby commands (#10479) · f110de4b
      Lu JJ authored
      Fixed a bug that used the `hincrbyfloat` or `hincrby` commands to make the field or value exceed the
      `hash_max_listpack_value` but did not change the object encoding of the hash structure.
      
      Add a length check for field and value, check the length of value first, if the length of value does not
      exceed `hash_max_listpack_value` then check the length of field.
      
      If the length of field or value is too long, it will reduce the efficiency of listpack, and the object encoding
      will become hashtable after AOF restart, so this is also to keep the same before and after AOF restart.
      f110de4b
    • judeng's avatar
      8a7049d3
    • Moti Cohen's avatar
      Stabilize Sentinel tests - refine failover-timeout & tilt-period (#10518) · e342bedc
      Moti Cohen authored
      Sentinel once in a while experience Sentinel TILT period or leader election 
      failure cycle. The problem is that those default timeout are too big and once 
      it happens, it breaks our tests.  Suggesting:
      - Reducing failover-timeout from 20 to 10sec (actually it is multiplied by 2 
        and reach 40sec of timeout) 
      - Modify tilt-period from default of 30sec to 5sec. When TILT period happens 
        it might lead to failover in our tests, and might cause also to failover cycle
        cycle failure.
      
      Sentinel tests should `wait_for_condition` up to 50seconds, where needed, 
      to be stable in case having single TILT period or failover failure cycle.
      
      In addition relax timing configuration for "manual failover" Sentinel test 
      (was modified several months ago as part of an effort to reduce tests runtime)
      e342bedc
    • Meir Shpilraien (Spielrein)'s avatar
      Functions: Move library meta data to be part of the library payload. (#10500) · ae020e3d
      Meir Shpilraien (Spielrein) authored
      ## Move library meta data to be part of the library payload.
      
      Following the discussion on https://github.com/redis/redis/issues/10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.**
      
      The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example:
      
      ```lua
      #!lua name=test
      redis.register_function('foo', function() return 1 end)
      ```
      
      The above script will run on the Lua engine and will create a library called `test`.
      
      ## API Changes (compare to 7.0 rc2)
      
      * `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`.
      * The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST`
      
      
      ## Breaking Changes (compare to 7.0 rc2)
      
      * Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line).
      * Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid.
      
      ## Notes
      
      * Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).
      ae020e3d
    • judeng's avatar
      delete obsolete REDISMODULE_EXPERIMENTAL_API define in module demos (#10527) · 9578b67e
      judeng authored
      This macro was recently removed from redismodule.h, so no longer needed.
      9578b67e
  14. 04 Apr, 2022 1 commit
  15. 03 Apr, 2022 1 commit
  16. 02 Apr, 2022 1 commit
    • Viktor Söderqvist's avatar
      Turn into replica on SETSLOT (#10489) · b53c7f2c
      Viktor Söderqvist authored
      * Fix race condition where node loses its last slot and turns into replica
      
      When a node has lost its last slot and finds out from the SETSLOT command
      before the cluster bus PONG from the new owner arrives. In this case, the
      node didn't turn itself into a replica of the new slot owner.
      
      This commit adds the same logic to the SETSLOT command as already exists
      for the cluster bus PONG processing.
      
      * Revert "Fix new / failing cluster slot migration test (#10482)"
      
      This reverts commit 0b21ef8d.
      
      In this test, the old slot owner finds out that it has lost its last
      slot in a nondeterministic way. Either the cluster bus PONG from the
      new slot owner and sometimes in a SETSLOT command from redis-cli. In
      both cases, the result should be the same and the old owner should
      turn itself into a replica of the new slot owner.
      b53c7f2c
  17. 31 Mar, 2022 3 commits
    • sundb's avatar
      Fix failing moduleconfigs tests and memory leak (#10501) · b8eb2a73
      sundb authored
      Fix global `strval` not reset to NULL after being freed, causing a crash on alpine
      (most likely because the dynamic library loader doesn't init globals on reload)
      By the way, fix the memory leak of using `RedisModule_Free` to free `RedisModuleString`,
      and add a corresponding test.
      b8eb2a73
    • Madelyn Olson's avatar
      Prevent replica failover during manual takeover test (#10499) · e81bd15e
      Madelyn Olson authored
      During 11-manual-takeover.tcl, if the killing of the instances happens
      too slowly, one of the replicas might be able to promote itself.
      I'm not sure why it was slow, but it was observed taking 6 seconds
      which is enough time to do an election.
      I was able to verify the error locally by adding a small delay (1 second)
      during ASAN CI. A fix is just to disable automated failover until all the
      nodes are confirmed dead.
      e81bd15e
    • Binbin's avatar
      Fix cluster slot migration test (#10495) · a3075ca4
      Binbin authored
      Fix three timing issues in the test
      a3075ca4
  18. 30 Mar, 2022 2 commits
    • Binbin's avatar
      Move restart_killed_instances and verify_sentinel_auto_discovery to utils (#10497) · 6075f506
      Binbin authored
      Create a utils.tcl in sentinel/tests/includes, and move two procs to it.
      Allow sentinel test 08-hostname-conf run on its own.
      6075f506
    • Nick Chun's avatar
      Module Configurations (#10285) · bda9d74d
      Nick Chun authored
      
      
      This feature adds the ability to add four different types (Bool, Numeric,
      String, Enum) of configurations to a module to be accessed via the redis
      config file, and the CONFIG command.
      
      **Configuration Names**:
      
      We impose a restriction that a module configuration always starts with the
      module name and contains a '.' followed by the config name. If a module passes
      "config1" as the name to a register function, it will be registered as MODULENAME.config1.
      
      **Configuration Persistence**:
      
      Module Configurations exist only as long as a module is loaded. If a module is
      unloaded, the configurations are removed.
      There is now also a minimal core API for removal of standardConfig objects
      from configs by name.
      
      **Get and Set Callbacks**:
      
      Storage of config values is owned by the module that registers them, and provides
      callbacks for Redis to access and manipulate the values.
      This is exposed through a GET and SET callback.
      
      The get callback returns a typed value of the config to redis. The callback takes
      the name of the configuration, and also a privdata pointer. Note that these only
      take the CONFIGNAME portion of the config, not the entire MODULENAME.CONFIGNAME.
      
      ```
       typedef RedisModuleString * (*RedisModuleConfigGetStringFunc)(const char *name, void *privdata);
       typedef long long (*RedisModuleConfigGetNumericFunc)(const char *name, void *privdata);
       typedef int (*RedisModuleConfigGetBoolFunc)(const char *name, void *privdata);
       typedef int (*RedisModuleConfigGetEnumFunc)(const char *name, void *privdata);
      ```
      
      Configs must also must specify a set callback, i.e. what to do on a CONFIG SET XYZ 123
      or when loading configurations from cli/.conf file matching these typedefs. *name* is
      again just the CONFIGNAME portion, *val* is the parsed value from the core,
      *privdata* is the registration time privdata pointer, and *err* is for providing errors to a client.
      
      ```
      typedef int (*RedisModuleConfigSetStringFunc)(const char *name, RedisModuleString *val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetNumericFunc)(const char *name, long long val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetBoolFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
      typedef int (*RedisModuleConfigSetEnumFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
      ```
      
      Modules can also specify an optional apply callback that will be called after
      value(s) have been set via CONFIG SET:
      
      ```
      typedef int (*RedisModuleConfigApplyFunc)(RedisModuleCtx *ctx, void *privdata, RedisModuleString **err);
      ```
      
      **Flags:**
      We expose 7 new flags to the module, which are used as part of the config registration.
      
      ```
      #define REDISMODULE_CONFIG_MODIFIABLE 0 /* This is the default for a module config. */
      #define REDISMODULE_CONFIG_IMMUTABLE (1ULL<<0) /* Can this value only be set at startup? */
      #define REDISMODULE_CONFIG_SENSITIVE (1ULL<<1) /* Does this value contain sensitive information */
      #define REDISMODULE_CONFIG_HIDDEN (1ULL<<4) /* This config is hidden in `config get <pattern>` (used for tests/debugging) */
      #define REDISMODULE_CONFIG_PROTECTED (1ULL<<5) /* Becomes immutable if enable-protected-configs is enabled. */
      #define REDISMODULE_CONFIG_DENY_LOADING (1ULL<<6) /* This config is forbidden during loading. */
      /* Numeric Specific Configs */
      #define REDISMODULE_CONFIG_MEMORY (1ULL<<7) /* Indicates if this value can be set as a memory value */
      ```
      
      **Module Registration APIs**:
      
      ```
      int (*RedisModule_RegisterBoolConfig)(RedisModuleCtx *ctx, char *name, int default_val, unsigned int flags, RedisModuleConfigGetBoolFunc getfn, RedisModuleConfigSetBoolFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterNumericConfig)(RedisModuleCtx *ctx, const char *name, long long default_val, unsigned int flags, long long min, long long max, RedisModuleConfigGetNumericFunc getfn, RedisModuleConfigSetNumericFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterStringConfig)(RedisModuleCtx *ctx, const char *name, const char *default_val, unsigned int flags, RedisModuleConfigGetStringFunc getfn, RedisModuleConfigSetStringFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_RegisterEnumConfig)(RedisModuleCtx *ctx, const char *name, int default_val, unsigned int flags, const char **enum_values, const int *int_values, int num_enum_vals, RedisModuleConfigGetEnumFunc getfn, RedisModuleConfigSetEnumFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
      int (*RedisModule_LoadConfigs)(RedisModuleCtx *ctx);
      ```
      
      The module name will be auto appended along with a "." to the front of the name of the config.
      
      **What RM_Register[...]Config does**:
      
      A RedisModule struct now keeps a list of ModuleConfig objects which look like:
      ```
      typedef struct ModuleConfig {
          sds name; /* Name of config without the module name appended to the front */
          void *privdata; /* Optional data passed into the module config callbacks */
          union get_fn { /* The get callback specificed by the module */
              RedisModuleConfigGetStringFunc get_string;
              RedisModuleConfigGetNumericFunc get_numeric;
              RedisModuleConfigGetBoolFunc get_bool;
              RedisModuleConfigGetEnumFunc get_enum;
          } get_fn;
          union set_fn { /* The set callback specified by the module */
              RedisModuleConfigSetStringFunc set_string;
              RedisModuleConfigSetNumericFunc set_numeric;
              RedisModuleConfigSetBoolFunc set_bool;
              RedisModuleConfigSetEnumFunc set_enum;
          } set_fn;
          RedisModuleConfigApplyFunc apply_fn;
          RedisModule *module;
      } ModuleConfig;
      ```
      It also registers a standardConfig in the configs array, with a pointer to the
      ModuleConfig object associated with it.
      
      **What happens on a CONFIG GET/SET MODULENAME.MODULECONFIG:**
      
      For CONFIG SET, we do the same parsing as is done in config.c and pass that
      as the argument to the module set callback. For CONFIG GET, we call the
      module get callback and return that value to config.c to return to a client.
      
      **CONFIG REWRITE**:
      
      Starting up a server with module configurations in a .conf file but no module load
      directive will fail. The flip side is also true, specifying a module load and a bunch
      of module configurations will load those configurations in using the module defined
      set callbacks on a RM_LoadConfigs call. Configs being rewritten works the same
      way as it does for standard configs, as the module has the ability to specify a
      default value. If a module is unloaded with configurations specified in the .conf file
      those configurations will be commented out from the .conf file on the next config rewrite.
      
      **RM_LoadConfigs:**
      
      `RedisModule_LoadConfigs(RedisModuleCtx *ctx);`
      
      This last API is used to make configs available within the onLoad() after they have
      been registered. The expected usage is that a module will register all of its configs,
      then call LoadConfigs to trigger all of the set callbacks, and then can error out if any
      of them were malformed. LoadConfigs will attempt to set all configs registered to
      either a .conf file argument/loadex argument or their default value if an argument is
      not specified. **LoadConfigs is a required function if configs are registered.
      ** Also note that LoadConfigs **does not** call the apply callbacks, but a module
      can do that directly after the LoadConfigs call.
      
      **New Command: MODULE LOADEX [CONFIG NAME VALUE] [ARGS ...]:**
      
      This command provides the ability to provide startup context information to a module.
      LOADEX stands for "load extended" similar to GETEX. Note that provided config
      names need the full MODULENAME.MODULECONFIG name. Any additional
      arguments a module might want are intended to be specified after ARGS.
      Everything after ARGS is passed to onLoad as RedisModuleString **argv.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      Co-authored-by: default avatarMadelyn Olson <matolson@amazon.com>
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      bda9d74d
  19. 28 Mar, 2022 2 commits
    • Oran Agra's avatar
      introduce MAX_D2STRING_CHARS instead of 128 const (#10487) · 14b19886
      Oran Agra authored
      There are a few places that use a hard coded const of 128 to allocate a buffer for d2string.
      Replace these with a clear macro.
      Note that In theory, converting double into string could take as much as nearly 400 chars,
      but since d2string uses `%g` and not `%f`, it won't pass some 40 chars.
      
      unrelated:
      restore some changes to auto generated commands.c that got accidentally reverted in #10293
      14b19886
    • Moti Cohen's avatar
      Fix sentinel test SDOWN is triggered by non-responding instance (#10484) · 63f77698
      Moti Cohen authored
      A timing issue of debug sleep master isn't long enough to ensure
      that master is down and let the test identify it. Replaced the code
      with suspend PID until verified master-is-down.
      63f77698
  20. 27 Mar, 2022 2 commits
    • Oran Agra's avatar
      Fix new / failing cluster slot migration test (#10482) · 0b21ef8d
      Oran Agra authored
      #10381 fixed an issue in `redis-cli --cluster reshard` that used to fail it (redis-cli) because
      of a race condition.
      the race condition is / was that when moving the last slot from a node, sometimes the PONG
      messages delivering the configuration change arrive to that node before the SETSLOT arrives
      to it, and it becomes a replica.
      other times the the SETSLOT arrive first, and then PONG **doesn't** demote it.
      
      **however**, the PR also added a new test that suffers from exactly the same race condition,
      and the tests started failing a lot.
      
      The fact is (if i understand it correctly), that this test (the one being deleted here), isn't related
      to the fix that PR fixed (which was to fix redis-cli).
      The race condition in the cluster code still happens, and as long as we don't solve it, there's
      no reason to test it.
      
      For now, even if my understandings are wrong, i'm gonna delete that failing test, since as far as
      i understand, #10381 didn't introduce any new risks for that matter (which are gonna be
      compromised by removing this check), this race existed since forever, and still exists, and the
      fact that redis-cli is now immune to it is still being tested.
      
      Additional work should be carried to fix it, and i live it for other PRs to handle.
      0b21ef8d
    • Moti Cohen's avatar
      Fix Sentinel reconnect test following ACL change (#10480) · 37d761ba
      Moti Cohen authored
      Replace condition with wait_for_condition On "Verify sentinel that restarted 
      failed to reconnect master after ACL change"
      
      The reason we reach it, is because the test is fast enough to modify ACL and
      test sentinel connection status with the server - before its scheduled operation
      got the chance to update connection status with the server:
      ```
      /* Perform scheduled operations for the specified Redis instance. */
      void sentinelHandleRedisInstance(sentinelRedisInstance *ri) {
          /* ========== MONITORING HALF ============ */
          /* Every kind of instance */
          sentinelReconnectInstance(ri);
      ```
      37d761ba
  21. 25 Mar, 2022 1 commit
    • zhaozhao.zz's avatar
      optimize(remove) usage of client's pending_querybuf (#10413) · 78bef6e1
      zhaozhao.zz authored
      To remove `pending_querybuf`, the key point is reusing `querybuf`, it means master client's `querybuf` is not only used to parse command, but also proxy to sub-replicas.
      
      1. add a new variable `repl_applied` for master client to record how many data applied (propagated via `replicationFeedStreamFromMasterStream()`) but not trimmed in `querybuf`.
      
      2. don't sdsrange `querybuf` in `commandProcessed()`, we trim it to `repl_applied` after the whole replication pipeline processed to avoid fragmented `sdsrange`. And here are some scenarios we cannot trim to `qb_pos`:
          * we don't receive complete command from master
          * master client blocked because of client pause
          * IO threads operate read, master client flagged with CLIENT_PENDING_COMMAND
      
          In these scenarios, `qb_pos` points to the part of the current command or the beginning of next command, and the current command is not applied yet, so the `repl_applied` is not equal to `qb_pos`.
      
      Some other notes:
      * Do not do big arg optimization on master client, since we can only sdsrange `querybuf` after data sent to replicas.
      * Set `qb_pos` and `repl_applied` to 0 when `freeClient` in `replicationCacheMaster`.
      * Rewrite `processPendingCommandsAndResetClient` to `processPendingCommandAndInputBuffer`, let `processInputBuffer` to be called successively after `processCommandAndResetClient`.
      78bef6e1
  22. 22 Mar, 2022 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Add new RM_Call flags for script mode, no writes, and error replies. (#10372) · f3855a09
      Meir Shpilraien (Spielrein) authored
      The PR extends RM_Call with 3 new capabilities using new flags that
      are given to RM_Call as part of the `fmt` argument.
      It aims to assist modules that are getting a list of commands to be
      executed from the user (not hard coded as part of the module logic),
      think of a module that implements a new scripting language...
      
      * `S` - Run the command in a script mode, this means that it will raise an
        error if a command which are not allowed inside a script (flaged with the
        `deny-script` flag) is invoked (like SHUTDOWN). In addition, on script mode,
        write commands are not allowed if there is not enough good replicas (as
        configured with `min-replicas-to-write`) and/or a disk error happened.
      
      * `W` - no writes mode, Redis will reject any command that is marked with `write`
        flag. Again can be useful to modules that implement a new scripting language
        and wants to prevent any write commands.
      
      * `E` - Return errors as RedisModuleCallReply. Today the errors that happened
        before the command was invoked (like unknown commands or acl error) return
        a NULL reply and set errno. This might be missing important information about
        the failure and it is also impossible to just pass the error to the user using
        RM_ReplyWithCallReply. This new flag allows you to get a RedisModuleCallReply
        object with the relevant error message and treat it as if it was an error that was
        raised by the command invocation.
      
      Tests were added to verify the new code paths.
      
      In addition small refactoring was done to share some code between modules,
      scripts, and `processCommand` function:
      1. `getAclErrorMessage` was added to `acl.c` to unified to log message extraction
        from the acl result
      2. `checkGoodReplicasStatus` was added to `replication.c` to check the status of
        good replicas. It is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and
        `processCommand`.
      3. `writeCommandsGetDiskErrorMessage` was added to `server.c` to get the error
        message on persistence failure. Again it is used on `scriptVerifyWriteCommandAllow`,
        `RM_Call`, and `processCommand`.
      f3855a09
  23. 21 Mar, 2022 3 commits