1. 21 Nov, 2024 2 commits
    • Moti Cohen's avatar
      modules API: Support register unprefixed config parameters (#13656) · 15563450
      Moti Cohen authored
      PR #10285 introduced support for modules to register four types of
      configurations — Bool, Numeric, String, and Enum. Accessible through the
      Redis config file and the CONFIG command.
      
      With this PR, it will be possible to register configuration parameters
      without automatically prefixing the parameter names. This provides
      greater flexibility in configuration naming, enabling, for instance,
      both `bf-initial-size` or `initial-size` to be defined in the module
      without automatically prefixing with `<MODULE-NAME>.`. In addition it
      will also be possible to create a single additional alias via the same
      API. This brings us another step closer to integrate modules into redis
      core.
      
      **Example:** Register a configuration parameter `bf-initial-size` with
      an alias `initial-size` without the automatic module name prefix, set
      with new `REDISMODULE_CONFIG_UNPREFIXED` flag:
      ```
      RedisModule_RegisterBoolConfig(ctx, "bf-initial-size|initial-size", default_val, optflags | REDISMODULE_CONFIG_UNPREFIXED, getfn, setfn, applyfn, privdata);
      ```
      # API changes
      Related functions that now support unprefixed configuration flag
      (`REDISMODULE_CONFIG_UNPREFIXED`) along with optional alias:
      ```
      RedisModule_RegisterBoolConfig
      RedisModule_RegisterEnumConfig
      RedisModule_RegisterNumericConfig
      RedisModule_RegisterStringConfig
      ```
      
      # Implementation Details:
      `config.c`: On load server configuration, at function
      `loadServerConfigFromString()`, it collects all unknown configurations
      into `module_configs_queue` dictionary. These may include valid module
      configurations or invalid ones. They will be validated later by
      `loadModuleConfigs()` against the configurations declared by the loaded
      module(s).
      `Module.c:` The `ModuleConfig` structure has been modified to store now:
      (1) Full configuration name (2) Alias (3) Unprefixed flag status -
      ensuring that configurations retain their original registration format
      when triggered in notifications.
      
      Added error printout:
      This change introduces an error printout for unresolved configurations,
      detailing each unresolved parameter detected during startup. The last
      line in the output existed prior to this change and has been retained to
      systems relies on it:
      ```
      595011:M 18 Nov 2024 08:26:23.616 # Unresolved Configuration(s) Detected:
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'bf-initiel-size 8'
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'search-sizex 32'
      595011:M 18 Nov 2024 08:26:23.616 # Module Configuration detected without loadmodule directive or no ApplyConfig call: aborting
      ```
      
      # Backward Compatibility:
      Existing modules will function without modification, as the new
      functionality only applies if REDISMODULE_CONFIG_UNPREFIXED is
      explicitly set.
      
      # Module vs. Core API Conflict Behavior
      The new API allows to modules loading duplication of same configuration
      name or same configuration alias, just like redis core configuration
      allows (i.e. the users sets two configs with a different value, but
      these two configs are actually the same one). Unlike redis core, given a
      name and its alias, it doesn't allow have both configuration on load. To
      implement it, it is required to modify DS `module_configs_queue` to
      reflect the order of their loading and later on, during
      `loadModuleConfigs()`, resolve pairs of names and aliases and which one
      is the last one to apply. "Relaxing" this limitation can be deferred to
      a future update if necessary, but for now, we error in this case.
      15563450
    • nafraf's avatar
      Fix module loadex command crash due to invalid config (#13653) · 5b84dc96
      nafraf authored
      Fix to https://github.com/redis/redis/issues/13650
      
      
      
      providing an invalid config to a module with datatype crashes when redis
      tries to unload the module due to the invalid config
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      5b84dc96
  2. 11 Nov, 2024 1 commit
    • Ozan Tezcan's avatar
      Print command tokens on a crash when hide-user-data-from-log is enabled (#13639) · 54038811
      Ozan Tezcan authored
      If `hide-user-data-from-log` config is enabled, we don't print client
      argv in the crashlog to avoid leaking user info.
      Though, debugging a crash becomes harder as we don't see the command
      arguments causing the crash.
      
      With this PR, we'll be printing command tokens to the log. As we have
      command tokens defined in json schema for each command, using this data,
      we can find tokens in the client argv.
      
      e.g. 
      `SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the
      log.
      
      Modules should introduce their command structure via
      `RM_SetCommandInfo()`.
      Then, on a crash we'll able to know module command tokens.
      54038811
  3. 19 Sep, 2024 1 commit
    • Moti Cohen's avatar
      Extend modules API to read also expired keys and subkeys (#13526) · 3a3cacfe
      Moti Cohen authored
      The PR extends `RedisModule_OpenKey`'s flags to include
      `REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired
      keys.
      
      It also allows to access expired subkeys. Currently relevant only for
      hash fields
      and has its impact on `RM_HashGet` and `RM_Scan`.
      3a3cacfe
  4. 04 Sep, 2024 1 commit
  5. 03 Sep, 2024 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Added new defrag API to allocate and free raw memory. (#13509) · d3d94ccf
      Meir Shpilraien (Spielrein) authored
      All the defrag allocations API expects to get a value and replace it, leaving the old value untouchable.
      In some cases a value might be shared between multiple keys, in such cases we can not simply replace
      it when the defrag callback is called.
      
      To allow support such use cases, the PR adds two new API's to the defrag API:
      
      1. `RM_DefragAllocRaw` - allocate memory base on a given size.
      2. `RM_DefragFreeRaw` - Free the given pointer.
      
      Those API's avoid using tcache so they operate just like `RM_DefragAlloc` but allows the user to split
      the allocation and the memory free operations into two stages and control when those happen.
      
      In addition the PR adds new API to allow the module to receive notifications when defrag start and end: `RM_RegisterDefragCallbacks`
      Those callbacks are the same as `RM_RegisterDefragFunc` but promised to be called and the start
      and the end of the defrag process.
      d3d94ccf
  6. 31 Jul, 2024 1 commit
  7. 17 Jul, 2024 1 commit
    • Oran Agra's avatar
      Fix external test hang in redis-cli test when run in a certain order (#13423) · a3319785
      Oran Agra authored
      When the tests are run against an external server in this order:
      `--single unit/introspection --single unit/moduleapi/blockonbackground
      --single integration/redis-cli`
      the test would hang when the "ASK redirect test" test attempts to create
      a listening socket (it fails, and then redis-cli itself hangs waiting
      for a non-responsive socket created by the introspection test).
      
      the reasons are:
      1. the blockedbackground test includes util.tcl and resets the
      `::last_port_attempted` variable
      2. the test in introspection didn't close the listening server, so it's
      still alive.
      3. find_available_port doesn't properly detect the busy port, and it
      thinks that the port is free even though it's busy.
      
      fixing all 3 of these problems, even though fixing just one would be
      enough to let the test pass.
      a3319785
  8. 26 Jun, 2024 1 commit
    • debing.sun's avatar
      Don't keep global replication buffer reference for replicas marked CLIENT_CLOSE_ASAP (#13363) · 52e12d8b
      debing.sun authored
      
      
      In certain situations, we might generate a large number of propagates
      (e.g., multi/exec, Lua script, or a single command generating tons of
      propagations) within an event loop.
      During the process of propagating to a replica, if the replica is
      disconnected(marked as CLIENT_CLOSE_ASAP) due to exceeding the output
      buffer limit, we should remove its reference to the global replication
      buffer to avoid the global replication buffer being unable to be
      properly trimmed due to being referenced.
      
      ---------
      Co-authored-by: default avataroranagra <oran@redislabs.com>
      52e12d8b
  9. 10 Jun, 2024 1 commit
    • Moti Cohen's avatar
      HFE - Avoid lazy expire if called by modules + cleanup (#13326) · ce121b92
      Moti Cohen authored
      Need to be carefull if called by modules since modules API allow to open
      and close key handler. We don't want to invalidate the handler
      underneath.
      
      * hashTypeExists(), hashTypeGetValueObject() - will return the logical
      state of the field. A flag will indicate noExpire.
      * RM_HashGet() - Will get NULL if the field expired. Fields won’t be
      deleted.
      * RM_ScanKey() - might return 0 items if all fields got expired. Fields
      won’t be deleted.
      * RM_HashSet() - If set, then override expired field. If delete, we can
      either delete or leave it to active-expiration. XX/NX - logically
      correct (Verify with tests).
      
      Nice to have (not implemented):
      * RedisModule_CloseKey() - We can local active-expire up-to 100 items. 
      
      Note:
      Length will be wrong to modules just like redis (Count expired fields).
      ce121b92
  10. 04 Jun, 2024 1 commit
  11. 12 Mar, 2024 1 commit
  12. 20 Feb, 2024 1 commit
    • Binbin's avatar
      Fix timing issue in blockedclient test (#13071) · 4e3be944
      Binbin authored
      We can see that the past time here happens to be busy_time_limit,
      causing the test to fail:
      ```
      [err]: RM_Call from blocked client in tests/unit/moduleapi/blockedclient.tcl
      Expected '50' to be more than '50' (context: type eval line 26 cmd {assert_morethan [expr [clock clicks -milliseconds]-$start] $busy_time_limit} proc ::test)
      ```
      
      It is reasonable for them to be equal, so equal is added here.
      It should be noted that in the previous `Busy module command` test,
      we also used assert_morethan_equal, so this should have been missed
      at the time.
      4e3be944
  13. 31 Jan, 2024 2 commits
    • Binbin's avatar
      Fix module assertion crash when timer and timeout are unlocked in the same event loop (#13015) · 6016973a
      Binbin authored
      When we use a timer to unblock a client in module, if the timer
      period and the block timeout are very close, they will unblock the
      client in the same event loop, and it will trigger the assertion.
      The reason is that in moduleBlockedClientTimedOut we will protect
      against re-processing, so we don't actually call updateStatsOnUnblock
      (see #12817), so we are not able to reset the c->duration. 
      
      The reason is unblockClientOnTimeout() didn't realize that bc had
      been unblocked. We add a function to the module to determine if bc
      is blocked, and then use it in unblockClientOnTimeout() to exit.
      
      There is the stack:
      ```
      beforeSleep
      blockedBeforeSleep
      handleBlockedClientsTimeout
      checkBlockedClientTimeout
      unblockClientOnTimeout
      unblockClient
      resetClient
      -- assertion, crash the server
      'c->duration == 0' is not true
      ```
      6016973a
    • Binbin's avatar
      Fix module unblock crash due to no timeout_callback (#13017) · 74a6e48a
      Binbin authored
      The block timeout is passed in the test case, but we do not pass
      in the timeout_callback, and it will crash when unlocking. In this
      case, in moduleBlockedClientTimedOut we will check timeout_callback.
      There is the stack:
      ```
      beforeSleep
      blockedBeforeSleep
      handleBlockedClientsTimeout
      checkBlockedClientTimeout
      unblockClientOnTimeout
      replyToBlockedClientTimedOut
      moduleBlockedClientTimedOut
      -- timeout_callback is NULL, invalidFunctionWasCalled
      bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
      ```
      74a6e48a
  14. 29 Jan, 2024 1 commit
  15. 09 Jan, 2024 1 commit
  16. 03 Jan, 2024 1 commit
    • Madelyn Olson's avatar
      Handle recursive serverAsserts and provide more information for recursive segfaults (#12857) · 068051e3
      Madelyn Olson authored
      This change is trying to make two failure modes a bit easier to deep dive:
      1. If a serverPanic or serverAssert occurs during the info (or module)
      printing, it will recursively panic, which is a lot of fun as it will
      just keep recursively printing. It will eventually stack overflow, but
      will generate a lot of text in the process.
      2. When a segfault happens during the segfault handler, no information
      is communicated other than it happened. This can be problematic because
      `info` may help diagnose the real issue, but without fixing the
      recursive crash it might be hard to get at that info.
      068051e3
  17. 11 Dec, 2023 1 commit
    • Binbin's avatar
      Fix delKeysInSlot server events are not executed inside an execution unit (#12745) · c85a9b78
      Binbin authored
      This is a follow-up fix to #12733. We need to apply the same changes to
      delKeysInSlot. Refer to #12733 for more details.
      
      This PR contains some other minor cleanups / improvements to the test
      suite and docs.
      It uses the postnotifications test module in a cluster mode test which
      revealed a leak in the test module (fixed).
      c85a9b78
  18. 27 Nov, 2023 1 commit
    • Binbin's avatar
      Un-register notification and server event when RedisModule_OnLoad fails (#12809) · d6f19539
      Binbin authored
      When we register notification or server event in RedisModule_OnLoad, but
      RedisModule_OnLoad eventually fails, triggering notification or server
      event
      will cause the server to crash.
      
      If the loading fails on a later stage of moduleLoad, we do call
      moduleUnload
      which handles all un-registration, but when it fails on the
      RedisModule_OnLoad
      call, we only un-register several specific things and these were
      missing:
      
      - moduleUnsubscribeNotifications
      - moduleUnregisterFilters
      - moduleUnsubscribeAllServerEvents
      
      Refactored the code to reuse the code from moduleUnload.
      
      Fixes #12808.
      d6f19539
  19. 23 Nov, 2023 1 commit
    • meiravgri's avatar
      Fix async safety in signal handlers (#12658) · 2e854bcc
      meiravgri authored
      see discussion from after https://github.com/redis/redis/pull/12453 was
      merged
      ----
      This PR replaces signals that are not considered async-signal-safe
      (AS-safe) with safe calls.
      
      #### **1. serverLog() and serverLogFromHandler()**
      `serverLog` uses unsafe calls. It was decided that we will **avoid**
      `serverLog` calls by the signal handlers when:
      * The signal is not fatal, such as SIGALRM. In these cases, we prefer
      using `serverLogFromHandler` which is the safe version of `serverLog`.
      Note they have different prompts:
      `serverLog`: `62220:M 26 Oct 2023 14:39:04.526 # <msg>`
      `serverLogFromHandler`: `62220:signal-handler (1698331136) <msg>`
      * The code was added recently. Calls to `serverLog` by the signal
      handler have been there ever since Redis exists and it hasn't caused
      problems so far. To avoid regression, from now we should use
      `serverLogFromHandler`
      
      #### **2. `snprintf` `fgets` and `strtoul`(base = 16) -------->
      `_safe_snprintf`, `fgets_async_signal_safe`, `string_to_hex`**
      The safe version of `snprintf` was taken from
      [here](https://github.com/twitter/twemcache/blob/8cfc4ca5e76ed936bd3786c8cc43ed47e7778c08/src/mc_util.c#L754)
      
      #### **3. fopen(), fgets(), fclose() --------> open(), read(), close()**
      
      #### **4. opendir(), readdir(), closedir() --------> open(),
      syscall(SYS_getdents64), close()**
      
      #### **5. Threads_mngr sync mechanisms**
      * waiting for the thread to generate stack trace: semaphore -------->
      busy-wait
      * `globals_rw_lock` was removed: as we are not using malloc and the
      semaphore anymore we don't need to protect `ThreadsManager_cleanups`.
      
      #### **6. Stacktraces buffer**
      The initial problem was that we were not able to safely call malloc
      within the signal handler.
      To solve that we created a buffer on the stack of `writeStacktraces` and
      saved it in a global pointer, assuming that under normal circumstances,
      the function `writeStacktraces` would complete before any thread
      attempted to write to it. However, **if threads lag behind, they might
      access this global pointer after it no longer belongs to the
      `writeStacktraces` stack, potentially corrupting memory.**
      To address this, various solutions were discussed
      [here](https://github.com/redis/redis/pull/12658#discussion_r1390442896)
      Eventually, we decided to **create a pipe** at server startup that will
      remain valid as long as the process is alive.
      We chose this solution due to its minimal memory usage, and since
      `write()` and `read()` are atomic operations. It ensures that stack
      traces from different threads won't mix.
      
      **The stacktraces collection process is now as  follows:**
      * Cleaning the pipe to eliminate writes of late threads from previous
      runs.
      * Each thread writes to the pipe its stacktrace
      * Waiting for all the threads to mark completion or until a timeout (2
      sec) is reached
      * Reading from the pipe to print the stacktraces.
      
      #### **7. Changes that were considered and eventually were dropped**
      * replace watchdog timer with a POSIX timer: 
      according to [settimer man](https://linux.die.net/man/2/setitimer)
      
      > POSIX.1-2008 marks getitimer() and setitimer() obsolete, recommending
      the use of the POSIX timers API
      ([timer_gettime](https://linux.die.net/man/2/timer_gettime)(2),
      [timer_settime](https://linux.die.net/man/2/timer_settime)(2), etc.)
      instead.
      
      However, although it is supposed to conform to POSIX std, POSIX timers
      API is not supported on Mac.
      You can take a look here at the Linux implementation:
      
      [here](https://github.com/redis/redis/commit/c7562ee13546e504977372fdf40d33c3f86775a5
      
      )
      To avoid messing up the code, and uncertainty regarding compatibility,
      it was decided to drop it for now.
      
      * avoid using sds (uses malloc) in logConfigDebugInfo
      It was considered to print config info instead of using sds, however
      apparently, `logConfigDebugInfo` does more than just print the sds, so
      it was decided this fix is out of this issue scope.
      
      #### **8. fix Signal mask check**
      The check `signum & sig_mask` intended to indicate whether the signal is
      blocked by the thread was incorrect. Actually, the bit position in the
      signal mask corresponds to the signal number. We fixed this by changing
      the condition to: `sig_mask & (1L << (sig_num - 1))`
      
      #### **9. Unrelated changes**
      both `fork.tcl `and `util.tcl` implemented a function called
      `count_log_message` expecting different parameters. This caused
      confusion when trying to run daily tests with additional test parameters
      to run a specific test.
      The `count_log_message` in `fork.tcl` was removed and the calls were
      replaced with calls to `count_log_message` located in `util.tcl`
      
      ---------
      Co-authored-by: default avatarOzan Tezcan <ozantezcan@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2e854bcc
  20. 08 Nov, 2023 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Before evicted and before expired server events are not executed inside an execution unit. (#12733) · 0ffb9d2e
      Meir Shpilraien (Spielrein) authored
      Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`.
      This new event allows the module to read the key data just before it is removed
      from the database (either deleted, expired, evicted, or overwritten).
      
      When the key is removed from the database, either by active expire or eviction.
      The new event was not called as part of an execution unit. This can cause an
      issue if the module registers a post notification job inside the event. This job will
      not be executed atomically with the expiration/eviction operation and will not
      replicated inside a Multi/Exec. Moreover, the post notification job will be executed
      right after the event where it is still not safe to perform any write operation, this will
      violate the promise that post notification job will be called atomically with the
      operation that triggered it and **only when it is safe to write**.
      
      This PR fixes the issue by wrapping each expiration/eviction of a key with an execution
      unit. This makes sure the entire operation will run atomically and all the post notification
      jobs will be executed at the end where it is safe to write.
      
      Tests were modified to verify the fix.
      0ffb9d2e
  21. 28 Sep, 2023 1 commit
    • guybe7's avatar
      WAITAOF: Update fsynced_reploff_pending even if there's nothing to fsync (#12622) · c2a4b784
      guybe7 authored
      The problem is that WAITAOF could have hang in case commands were
      propagated only to replicas.
      This can happen if a module uses RM_Call with the REDISMODULE_ARGV_NO_AOF flag.
      In that case, master_repl_offset would increase, but there would be nothing to fsync, so
      in the absence of other traffic, fsynced_reploff_pending would stay the static, and WAITAOF can hang.
      
      This commit updates fsynced_reploff_pending to the latest offset in flushAppendOnlyFile in case
      there's nothing to fsync. i.e. in case it's behind because of the above mentions case it'll be refreshed
      and release the WAITAOF.
      
      Other changes:
      Fix a race in wait.tcl (client getting blocked vs. the fsync thread)
      c2a4b784
  22. 30 Aug, 2023 1 commit
    • Roshan Khatri's avatar
      Allows modules to declare new ACL categories. (#12486) · 75199605
      Roshan Khatri authored
      
      
      This PR adds a new Module API int RM_AddACLCategory(RedisModuleCtx *ctx, const char *category_name) to add a new ACL command category.
      
      Here, we initialize the ACLCommandCategories array by allocating space for 64 categories and duplicate the 21 default categories from the predefined array 'ACLDefaultCommandCategories' into the ACLCommandCategories array while ACL initialization. Valid ACL category names can only contain alphanumeric characters, underscores, and dashes.
      
      The API when called, checks for the onload flag, category name validity, and for duplicate category name if present. If the conditions are satisfied, the API adds the new category to the trailing end of the ACLCommandCategories array and assigns the acl_categories flag bit according to the index at which the category is added.
      
      If any error is encountered the errno is set accordingly by the API.
      
      ---------
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      75199605
  23. 27 Jun, 2023 1 commit
    • judeng's avatar
      improve performance for scan command when matching pattern or data type (#12209) · 07ed0eaf
      judeng authored
      
      
      Optimized the performance of the SCAN command in a few ways:
      1. Move the key filtering (by MATCH pattern) in the scan callback,
        so as to avoid collecting them for later filtering.
      2. Reduce a many memory allocations and copying (use a reference
        to the original sds, instead of creating an robj, an excessive 2 mallocs
        and one string duplication)
      3. Compare TYPE filter directly (as integers), instead of inefficient string
        compare per key.
      4. fixed a small bug: when scan zset and hash types, maxiterations uses
        a more accurate number to avoid wrong double maxiterations.
      
      Changes **postponed** for a later version (8.0):
      1. Prepare to move the TYPE filtering to the scan callback as well. this was
        put on hold since it has side effects that can be considered a breaking
        change, which is that we will not attempt to do lazy expire (delete) a key
        that was filtered by not matching the TYPE (changing it would mean TYPE filter
        starts behaving the same as MATCH filter already does in that respect). 
      2. when the specified key TYPE filter is an unknown type, server will reply a error
        immediately instead of doing a full scan that comes back empty handed. 
      
      Benchmark result:
      For different scenarios, we obtained about 30% or more performance improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      07ed0eaf
  24. 25 Jun, 2023 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Fix use after free on blocking RM_Call. (#12342) · 153f8f08
      Meir Shpilraien (Spielrein) authored
      blocking RM_Call was introduced on: #11568, It allows a module to perform
      blocking commands and get the reply asynchronously.If the command gets
      block, a special promise CallReply is returned that allow to set the unblock
      handler. The unblock handler will be called when the command invocation
      finish and it gets, as input, the command real reply.
      
      The issue was that the real CallReply was created using a stack allocated
      RedisModuleCtx which is no longer available after the unblock handler finishes.
      So if the module keeps the CallReply after the unblock handler finished, the
      CallReply holds a pointer to invalid memory and will try to access it when the
      CallReply will be released.
      
      The solution is to create the CallReply with a NULL context to make it totally
      detached and can be freed freely when the module wants.
      
      Test was added to cover this case, running the test with valgrind before the
      fix shows the use after free error. With the fix, there are no valgrind errors.
      
      unrelated: adding a missing `$rd close` in many tests in that file.
      153f8f08
  25. 22 Jun, 2023 1 commit
    • guybe7's avatar
      Modules: Unblock from within a timer coverage (#12337) · 32301999
      guybe7 authored
      Apart from adding the missing coverage, this PR also adds `blockedBeforeSleep`
      that gathers all block-related functions from `beforeSleep`
      
      The order inside `blockedBeforeSleep` is different: now `handleClientsBlockedOnKeys`
      (which may unblock clients) is called before `processUnblockedClients` (which handles
      unblocked clients).
      It makes sense to have this order.
      
      There are no visible effects of the wrong ordering, except some cleanups of the now-unblocked
      client would have  happen in the next `beforeSleep` (will now happen in the current one)
      
      The reason we even got into it is because i triggers an assertion in logresreq.c (breaking
      the assumption that `unblockClient` is called **before** actually flushing the reply to the socket):
      `handleClientsBlockedOnKeys` is called, then it calls `moduleUnblockClientOnKey`, which calls
      `moduleUnblockClient`, which adds the client to `moduleUnblockedClients` back to `beforeSleep`,
      we call `handleClientsWithPendingWritesUsingThreads`, it writes the data of buf to the client, so
      `client->bufpos` became 0
      On the next `beforeSleep`, we call `moduleHandleBlockedClients`, which calls `unblockClient`,
      which calls `reqresAppendResponse`, triggering the assert. (because the `bufpos` is 0) - see https://github.com/redis/redis/pull/12301#discussion_r1226386716
      32301999
  26. 20 Jun, 2023 3 commits
    • guybe7's avatar
      Align RM_ReplyWithErrorFormat with RM_ReplyWithError (#12321) · 20fa1560
      guybe7 authored
      Introduced by https://github.com/redis/redis/pull/11923 (Redis 7.2 RC2)
      
      It's very weird and counterintuitive that `RM_ReplyWithError` requires the error-code
      **without** a hyphen while `RM_ReplyWithErrorFormat` requires either the error-code
      **with** a hyphen or no error-code at all
      ```
      RedisModule_ReplyWithError(ctx, "BLA bla bla");
      ```
      vs.
      ```
      RedisModule_ReplyWithErrorFormat(ctx, "-BLA %s", "bla bla");
      ```
      
      This commit aligns RM_ReplyWithErrorFormat to behvae like RM_ReplyWithError.
      it's a breaking changes but it's done before 7.2 goes GA.
      20fa1560
    • Oran Agra's avatar
      Fix broken protocol when PUBLISH emits local push inside MULTI (#12326) · 8ad8f0f9
      Oran Agra authored
      When a connection that's subscribe to a channel emits PUBLISH inside MULTI-EXEC,
      the push notification messes up the EXEC response.
      
      e.g. MULTI, PING, PUSH foo bar, PING, EXEC
      the EXEC's response will contain: PONG, {message foo bar}, 1. and the second PONG
      will be delivered outside the EXEC's response.
      
      Additionally, this PR changes the order of responses in case of a plain PUBLISH (when
      the current client also subscribed to it), by delivering the push after the command's
      response instead of before it.
      This also affects modules calling RM_PublishMessage in a similar way, so that we don't
      run the risk of getting that push mixed together with the module command's response.
      8ad8f0f9
    • Shaya Potter's avatar
      Add ability for modules to know which client is being cmd filtered (#12219) · 07316f16
      Shaya Potter authored
      Adds API
      - RedisModule_CommandFilterGetClientId()
      
      Includes addition to commandfilter test module to validate that it works
      by performing the same command from 2 different clients
      07316f16
  27. 28 May, 2023 1 commit
    • Oran Agra's avatar
      Fix WAIT for clients being blocked in a module command (#12220) · 6117f288
      Oran Agra authored
      So far clients being blocked and unblocked by a module command would
      update the c->woff variable and so WAIT was ineffective and got released
      without waiting for the command actions to propagate.
      
      This seems to have existed since forever, but not for RM_BlockClientOnKeys.
      
      It is problematic though to know if the module did or didn't propagate
      anything in that command, so for now, instead of adding an API, we'll
      just update the woff to the latest offset when unblocking, this will
      cause the client to possibly wait excessively, but that's not that bad.
      6117f288
  28. 07 May, 2023 1 commit
    • sundb's avatar
      Delete empty key if fails after moduleCreateEmptyKey() in module (#12129) · ce5f4ea3
      sundb authored
      When `RM_ZsetAdd()`/`RM_ZsetIncrby()`/`RM_StreamAdd()` fails, if a new key happens to 
      be created using `moduleCreateEmptyKey()`, we should clean up the empty key.
      
      ## Test
      1) Add new module commands(`zset.add` and `zset.incrby`) to cover  `RM_ZsetAdd()`/`RM_ZsetIncrby()`.
      2) Add a large-memory test to cover `RM_StreamAdd()`.
      ce5f4ea3
  29. 18 Apr, 2023 1 commit
  30. 12 Apr, 2023 2 commits
    • Binbin's avatar
      Add RM_ReplyWithErrorFormat that can support format (#11923) · bfec2d70
      Binbin authored
      * Add RM_ReplyWithErrorFormat that can support format
      
      Reply with the error create from a printf format and arguments.
      
      If the error code is already passed in the string 'fmt', the error
      code provided is used, otherwise the string "-ERR " for the generic
      error code is automatically added.
      
      The usage is, for example:
          RedisModule_ReplyWithErrorFormat(ctx, "An error: %s", "foo");
          RedisModule_ReplyWithErrorFormat(ctx, "-WRONGTYPE Wrong Type: %s", "foo");
      
      The function always returns REDISMODULE_OK.
      bfec2d70
    • Oran Agra's avatar
      Attempt to solve MacOS CI issues in GH Actions (#12013) · 997fa41e
      Oran Agra authored
      The MacOS CI in github actions often hangs without any logs. GH argues that
      it's due to resource utilization, either running out of disk space, memory, or CPU
      starvation, and thus the runner is terminated.
      
      This PR contains multiple attempts to resolve this:
      1. introducing pause_process instead of SIGSTOP, which waits for the process
        to stop before resuming the test, possibly resolving race conditions in some tests,
        this was a suspect since there was one test that could result in an infinite loop in that
       case, in practice this didn't help, but still a good idea to keep.
      2. disable the `save` config in many tests that don't need it, specifically ones that use
        heavy writes and could create large files.
      3. change the `populate` proc to use short pipeline rather than an infinite one.
      4. use `--clients 1` in the macos CI so that we don't risk running multiple resource
        demanding tests in parallel.
      5. enable `--verbose` to be repeated to elevate verbosity and print more info to stdout
        when a test or a server starts.
      997fa41e
  31. 09 Apr, 2023 1 commit
    • Ozan Tezcan's avatar
      Add RM_RdbLoad and RM_RdbSave module API functions (#11852) · e55568ed
      Ozan Tezcan authored
      Add `RM_RdbLoad()` and `RM_RdbSave()` to load/save RDB files from the module API. 
      
      In our use case, we have our clustering implementation as a module. As part of this
      implementation, the module needs to trigger RDB save operation at specific points.
      Also, this module delivers RDB files to other nodes (not using Redis' replication).
      When a node receives an RDB file, it should be able to load the RDB. Currently,
      there is no module API to save/load RDB files. 
      
      
      This PR adds four new APIs:
      ```c
      RedisModuleRdbStream *RM_RdbStreamCreateFromFile(const char *filename);
      void RM_RdbStreamFree(RedisModuleRdbStream *stream);
      
      int RM_RdbLoad(RedisModuleCtx *ctx, RedisModuleRdbStream *stream, int flags);
      int RM_RdbSave(RedisModuleCtx *ctx, RedisModuleRdbStream *stream, int flags);
      ```
      
      The first step is to create a `RedisModuleRdbStream` object. This PR provides a function to
      create RedisModuleRdbStream from the filename. (You can load/save RDB with the filename).
      In the future, this API can be extended if needed: 
      e.g., `RM_RdbStreamCreateFromFd()`, `RM_RdbStreamCreateFromSocket()` to save/load
      RDB from an `fd` or a `socket`. 
      
      
      Usage:
      ```c
      /* Save RDB */
      RedisModuleRdbStream *stream = RedisModule_RdbStreamCreateFromFile("example.rdb");
      RedisModule_RdbSave(ctx, stream, 0);
      RedisModule_RdbStreamFree(stream);
      
      /* Load RDB */
      RedisModuleRdbStream *stream = RedisModule_RdbStreamCreateFromFile("example.rdb");
      RedisModule_RdbLoad(ctx, stream, 0);
      RedisModule_RdbStreamFree(stream);
      ```
      e55568ed
  32. 21 Mar, 2023 1 commit
    • Roshan Khatri's avatar
      Module commands to have ACL categories. (#11708) · 6948daca
      Roshan Khatri authored
      
      
      This allows modules to register commands to existing ACL categories and blocks the creation of [sub]commands, datatypes and registering the configs outside of the OnLoad function.
      
      For allowing modules to register commands to existing ACL categories,
      This PR implements a new API int RM_SetCommandACLCategories() which takes a pointer to a RedisModuleCommand and a C string aclflags containing the set of space separated ACL categories.
      Example, 'write slow' marks the command as part of the write and slow ACL categories.
      
      The C string aclflags is tokenized by implementing a helper function categoryFlagsFromString(). Theses tokens are matched and the corresponding ACL categories flags are set by a helper function matchAclCategoriesFlags. The helper function categoryFlagsFromString() returns the corresponding categories_flags or returns -1 if some token not processed correctly.
      
      If the module contains commands which are registered to existing ACL categories, the number of [sub]commands are tracked by num_commands_with_acl_categories in struct RedisModule. Further, the allowed command bit-map of the existing users are recomputed from the command_rules list, by implementing a function called ACLRecomputeCommandBitsFromCommandRulesAllUsers() for the existing users to have access to the module commands on runtime.
      
      ## Breaking change
      This change requires that registering commands and subcommands only occur during a modules "OnLoad" function, in order to allow efficient recompilation of ACL bits. We also chose to block registering configs and types, since we believe it's only valid for those to be created during onLoad. We check for this onload flag in struct RedisModule to check if the call is made from the OnLoad function.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      6948daca
  33. 20 Mar, 2023 1 commit
    • Shaya Potter's avatar
      Don't run command filter on blocked command reprocessing (#11895) · 6cf8fc08
      Shaya Potter authored
      
      
      Previously we would run the module command filters even upon blocked
      command reprocessing.  This could modify the command, and it's args.
      This is irrelevant in the context of a command being reprocessed (it already
      went through the filters), as well as breaks the crashed command lookup
      that exists in the case of a reprocessed command.
      
      fixes #11894.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      6cf8fc08
  34. 16 Mar, 2023 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Support for RM_Call on blocking commands (#11568) · d0da0a6a
      Meir Shpilraien (Spielrein) authored
      Allow running blocking commands from within a module using `RM_Call`.
      
      Today, when `RM_Call` is used, the fake client that is used to run command
      is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command
      that it is not allowed to block the client and in case it needs to block, it must
      fallback to some alternative (either return error or perform some default behavior).
      For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block.
      
      All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including
      module commands). When the command invocation finished, Redis asserts that
      the client was not blocked.
      
      This PR introduces the ability to call blocking command using `RM_Call` by
      passing a callback that will be called when the client will get unblocked.
      In order to do that, the user must explicitly say that he allow to perform blocking
      command by passing a new format specifier argument, `K`, to the `RM_Call`
      function. This new flag will tell Redis that it is allow to run blocking command
      and block the client. In case the command got blocked, Redis will return a new
      type of call reply (`REDISMODULE_REPLY_PROMISE`). This call reply indicates
      that the command got blocked and the user can set the on_unblocked handler using
      `RM_CallReplyPromiseSetUnblockHandler`.
      
      When clients gets unblocked, it eventually reaches `processUnblockedClients` function.
      This is where we check if the client is a fake module client and if it is, we call the unblock
      callback instead of performing the usual unblock operations.
      
      **Notice**: `RM_CallReplyPromiseSetUnblockHandler` must be called atomically
      along side the command invocation (without releasing the Redis lock in between).
      In addition, unlike other CallReply types, the promise call reply must be released
      by the module when the Redis GIL is acquired.
      
      The module can abort the execution on the blocking command (if it was not yet
      executed) using `RM_CallReplyPromiseAbort`. the API will return `REDISMODULE_OK`
      on success and `REDISMODULE_ERR` if the operation is already executed.
      **Notice** that in case of misbehave module, Abort might finished successfully but the
      operation will not really be aborted. This can only happened if the module do not respect
      the disconnect callback of the blocked client. 
      For pure Redis commands this can not happened.
      
      ### Atomicity Guarantees
      
      The API promise that the unblock handler will run atomically as an execution unit.
      This means that all the operation performed on the unblock handler will be wrapped
      with a multi exec transaction when replicated to the replica and AOF.
      The API **do not** grantee any other atomicity properties such as when the unblock
      handler will be called. This gives us the flexibility to strengthen the grantees (or not)
      in the future if we will decide that we need a better guarantees.
      
      That said, the implementation **does** provide a better guarantees when performing
      pure Redis blocking command like `BLPOP`. In this case the unblock handler will run
      atomically with the operation that got unblocked (for example, in case of `BLPOP`, the
      unblock handler will run atomically with the `LPOP` operation that run when the command
      got unblocked). This is an implementation detail that might be change in the future and the
      module writer should not count on that.
      
      ### Calling blocking commands while running on script mode (`S`)
      
      `RM_Call` script mode (`S`) was introduced on #0372. It is used for usecases where the
      command that was invoked on `RM_Call` comes from a user input and we want to make
      sure the user will not run dangerous commands like `shutdown`. Some command, such
      as `BLPOP`, are marked with `NO_SCRIPT` flag, which means they will not be allowed on
      script mode. Those commands are marked with  `NO_SCRIPT` just because they are
      blocking commands and not because they are dangerous. Now that we can run blocking
      commands on RM_Call, there is no real reason not to allow such commands on script mode.
      
      The underline problem is that the `NO_SCRIPT` flag is abused to also mark some of the
      blocking commands (notice that those commands know not to block the client if it is not
      allowed to do so, and have a fallback logic to such cases. So even if those commands
      were not marked with `NO_SCRIPT` flag, it would not harm Redis, and today we can
      already run those commands within multi exec).
      
      In addition, not all blocking commands are marked with `NO_SCRIPT` flag, for example
      `blmpop` are not marked and can run from within a script.
      
      Those facts shows that there are some ambiguity about the meaning of the `NO_SCRIPT`
      flag, and its not fully clear where it should be use.
      
      The PR suggest that blocking commands should not be marked with `NO_SCRIPT` flag,
      those commands should handle `CLIENT_DENY_BLOCKING` flag and only block when
      it's safe (like they already does today). To achieve that, the PR removes the `NO_SCRIPT`
      flag from the following commands:
      * `blmove`
      * `blpop`
      * `brpop`
      * `brpoplpush`
      * `bzpopmax`
      * `bzpopmin`
      * `wait`
      
      This might be considered a breaking change as now, on scripts, instead of getting
      `command is not allowed from script` error, the user will get some fallback behavior
      base on the command implementation. That said, the change matches the behavior
      of scripts and multi exec with respect to those commands and allow running them on
      `RM_Call` even when script mode is used.
      
      ### Additional RedisModule API and changes
      
      * `RM_BlockClientSetPrivateData` - Set private data on the blocked client without the
        need to unblock the client. This allows up to set the promise CallReply as the private
        data of the blocked client and abort it if the client gets disconnected.
      * `RM_BlockClientGetPrivateData` - Return the current private data set on a blocked client.
        We need it so we will have access to this private data on the disconnect callback.
      * On RM_Call, the returned reply will be added to the auto memory context only if auto
        memory is enabled, this allows us to keep the call reply for longer time then the context
        lifetime and does not force an unneeded borrow relationship between the CallReply and
        the RedisModuleContext.
      d0da0a6a
  35. 15 Mar, 2023 1 commit
    • KarthikSubbarao's avatar
      Custom authentication for Modules (#11659) · f8a5a4f7
      KarthikSubbarao authored
      
      
      This change adds new module callbacks that can override the default password based authentication associated with ACLs. With this, Modules can register auth callbacks through which they can implement their own Authentication logic. When `AUTH` and `HELLO AUTH ...` commands are used, Module based authentication is attempted and then normal password based authentication is attempted if needed.
      The new Module APIs added in this PR are - `RM_RegisterCustomAuthCallback` and `RM_BlockClientOnAuth` and `RedisModule_ACLAddLogEntryByUserName `.
      
      Module based authentication will be attempted for all Redis users (created through the ACL SETUSER cmd or through Module APIs) even if the Redis user does not exist at the time of the command. This gives a chance for the Module to create the RedisModule user and then authenticate via the RedisModule API - from the custom auth callback.
      
      For the AUTH command, we will support both variations - `AUTH <username> <password>` and `AUTH <password>`. In case of the `AUTH <password>` variation, the custom auth callbacks are triggered with “default” as the username and password as what is provided.
      
      
      ### RedisModule_RegisterCustomAuthCallback
      ```
      void RM_RegisterCustomAuthCallback(RedisModuleCtx *ctx, RedisModuleCustomAuthCallback cb) {
      ```
      This API registers a callback to execute to prior to normal password based authentication. Multiple callbacks can be registered across different modules. These callbacks are responsible for either handling the authentication, each authenticating the user or explicitly denying, or deferring it to other authentication mechanisms. Callbacks are triggered in the order they were registered. When a Module is unloaded, all the auth callbacks registered by it are unregistered. The callbacks are attempted, in the order of most recently registered callbacks, when the AUTH/HELLO (with AUTH field is provided) commands are called. The callbacks will be called with a module context along with a username and a password, and are expected to take one of the following actions:
      
       (1) Authenticate - Use the RM_Authenticate* API successfully and return `REDISMODULE_AUTH_HANDLED`. This will immediately end the auth chain as successful and add the OK reply.
      (2) Block a client on authentication - Use the `RM_BlockClientOnAuth` API and return `REDISMODULE_AUTH_HANDLED`. Here, the client will be blocked until the `RM_UnblockClient `API is used which will trigger the auth reply callback (provided earlier through the `RM_BlockClientOnAuth`). In this reply callback, the Module should authenticate, deny or skip handling authentication.
      (3) Deny Authentication - Return `REDISMODULE_AUTH_HANDLED` without authenticating or blocking the client. Optionally, `err` can be set to a custom error message. This will immediately end the auth chain as unsuccessful and add the ERR reply.
      (4) Skip handling Authentication - Return `REDISMODULE_AUTH_NOT_HANDLED` without blocking the client. This will allow the engine to attempt the next custom auth callback.
      
      If none of the callbacks authenticate or deny auth, then password based auth is attempted and will authenticate or add failure logs and reply to the clients accordingly.
      
      ### RedisModule_BlockClientOnAuth
      ```
      RedisModuleBlockedClient *RM_BlockClientOnAuth(RedisModuleCtx *ctx, RedisModuleCustomAuthCallback reply_callback,
                                                     void (*free_privdata)(RedisModuleCtx*,void*))
      ```
      This API can only be used from a Module from the custom auth callback. If a client is not in the middle of custom module based authentication, ERROR is returned. Otherwise, the client is blocked and the `RedisModule_BlockedClient` is returned similar to the `RedisModule_BlockClient` API.
      
      ### RedisModule_ACLAddLogEntryByUserName
      ```
      int RM_ACLAddLogEntryByUserName(RedisModuleCtx *ctx, RedisModuleString *username, RedisModuleString *object, RedisModuleACLLogEntryReason reason)
      ```
      Adds a new entry in the ACL log with the `username` RedisModuleString provided. This simplifies the Module usage because now, developers do not need to create a Module User just to add an error ACL Log entry. Aside from accepting username (RedisModuleString) instead of a RedisModuleUser, it is the same as the existing `RedisModule_ACLAddLogEntry` API.
      
      
      ### Breaking changes
      - HELLO command - Clients can now only set the client name and RESP protocol from the `HELLO` command if they are authenticated. Also, we now finish command arg validation first and return early with a ERR reply if any arg is invalid. This is to avoid mutating the client name / RESP from a command that would have failed on invalid arguments.
      
      ### Notable behaviors
      - Module unblocking - Now, we will not allow Modules to block the client from inside the context of a reply callback (triggered from the Module unblock flow `moduleHandleBlockedClients`).
      
      ---------
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      f8a5a4f7