1. 28 Feb, 2023 1 commit
    • uriyage's avatar
      Optimization: sdsRemoveFreeSpace to avoid realloc on noop (#11766) · f084778c
      uriyage authored
      
      
      In #7875 (Redis 6.2), we changed the sds alloc to be the usable allocation
      size in order to:
      
      > reduce the need for realloc calls by making the sds implicitly take over
      the internal fragmentation
      
      This change was done most sds functions, excluding `sdsRemoveFreeSpace` and
      `sdsResize`, the reason is that in some places (e.g. clientsCronResizeQueryBuffer)
      we call sdsRemoveFreeSpace when we see excessive free space and want to trim it.
      so if we don't trim it exactly to size, the caller may still see excessive free space and
      call it again and again.
      
      However, this resulted in some excessive calls to realloc, even when there's no need
      and it's gonna be a no-op (e.g. when reducing 15 bytes allocation to 13).
      
      It turns out that a call for realloc with jemalloc can be expensive even if it ends up
      doing nothing, so this PR adds a check using `je_nallocx`, which is cheap to avoid
      the call for realloc.
      
      in addition to that this PR unifies sdsResize and sdsRemoveFreeSpace into common
      code. the difference between them was that sdsResize would avoid using SDS_TYPE_5,
      since it want to keep the string ready to be resized again, while sdsRemoveFreeSpace
      would permit using SDS_TYPE_5 and get an optimal memory consumption.
      now both methods take a `would_regrow` argument that makes it more explicit.
      
      the only actual impact of that is that in clientsCronResizeQueryBuffer we call both sdsResize
      and sdsRemoveFreeSpace for in different cases, and we now prevent the use of SDS_TYPE_5 in both.
      
      The new test that was added to cover this concern used to pass before this PR as well,
      this PR is just a performance optimization and cleanup.
      
      Benchmark:
      `redis-benchmark -c 100 -t set  -d 512 -P 10  -n  100000000`
      on i7-9850H with jemalloc, shows improvement from 1021k ops/sec to 1067k (average of 3 runs).
      some 4.5% improvement.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 46393f98)
      (cherry picked from commit b12eeccddd9318a5d97a5aee2dad88999dfad53f)
      f084778c
  2. 12 Dec, 2022 2 commits
  3. 27 Apr, 2022 2 commits
    • Vitah Lin's avatar
      Fix memory leak in RM_StreamIteratorStop and moduleFreeKeyIterator (#10353) · 2643c691
      Vitah Lin authored
      * Fix memory leak in RM_StreamIteratorStop
      * Fix memory leak in moduleFreeKeyIterator
      
      (cherry picked from commit dff153ff)
      2643c691
    • Oran Agra's avatar
      Fix and improve module error reply statistics (#10278) · d6a8e64e
      Oran Agra authored
      This PR handles several aspects
      1. Calls to RM_ReplyWithError from thread safe contexts don't violate thread safety.
      2. Errors returning from RM_Call to the module aren't counted in the statistics (they
        might be handled silently by the module)
      3. When a module propagates a reply it got from RM_Call to it's client, then the error
        statistics are counted.
      
      This is done by:
      1. When appending an error reply to the output buffer, we avoid updating the global
        error statistics, instead we cache that error in a deferred list in the client struct.
      2. When creating a RedisModuleCallReply object, the deferred error list is moved from
        the client into that object.
      3. when a module calls RM_ReplyWithCallReply we copy the deferred replies to the dest
        client (if that's a real client, then that's when the error statistics are updated to the server)
      
      Note about RM_ReplyWithCallReply: if the original reply had an array with errors, and the module
      replied with just a portion of the original reply, and not the entire reply, the errors are currently not
      propagated and the errors stats will not get propagated.
      
      Fix #10180
      
      (cherry picked from commit b099889a)
      d6a8e64e
  4. 04 Oct, 2021 1 commit
    • Oran Agra's avatar
      Fix ziplist and listpack overflows and truncations (CVE-2021-32627, CVE-2021-32628) · 2775a352
      Oran Agra authored
      - fix possible heap corruption in ziplist and listpack resulting by trying to
        allocate more than the maximum size of 4GB.
      - prevent ziplist (hash and zset) from reaching size of above 1GB, will be
        converted to HT encoding, that's not a useful size.
      - prevent listpack (stream) from reaching size of above 1GB.
      - XADD will start a new listpack if the new record may cause the previous
        listpack to grow over 1GB.
      - XADD will respond with an error if a single stream record is over 1GB
      - List type (ziplist in quicklist) was truncating strings that were over 4GB,
        now it'll respond with an error.
      2775a352
  5. 21 Jul, 2021 5 commits
    • Oran Agra's avatar
      Adjustments to recent RM_StringTruncate fix (#3718) (#9125) · 37b0f361
      Oran Agra authored
      - Introduce a new sdssubstr api as a building block for sdsrange.
        The API of sdsrange is many times hard to work with and also has
        corner case that cause bugs. sdsrange is easy to work with and also
        simplifies the implementation of sdsrange.
      - Revert the fix to RM_StringTruncate and just use sdssubstr instead of
        sdsrange.
      - Solve valgrind warnings from the new tests introduced by the previous
        PR.
      
      (cherry picked from commit ae418eca)
      37b0f361
    • Huang Zhw's avatar
      Fix missing separator in module info line (usedby and using lists) (#9241) · 68661171
      Huang Zhw authored
      
      
      Fix module info genModulesInfoStringRenderModulesList lack separator when there's more than one module in the list.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit 1895e134)
      68661171
    • Yossi Gottlieb's avatar
      Fix CLIENT UNBLOCK crashing modules. (#9167) · 79fa5618
      Yossi Gottlieb authored
      Modules that use background threads with thread safe contexts are likely
      to use RM_BlockClient() without a timeout function, because they do not
      set up a timeout.
      
      Before this commit, `CLIENT UNBLOCK` would result with a crash as the
      `NULL` timeout callback is called. Beyond just crashing, this is also
      logically wrong as it may throw the module into an unexpected client
      state.
      
      This commits makes `CLIENT UNBLOCK` on such clients behave the same as
      any other client that is not in a blocked state and therefore cannot be
      unblocked.
      
      (cherry picked from commit aa139e2f)
      79fa5618
    • Evan's avatar
      modules: Add newlen == 0 handling to RM_StringTruncate (#3717) (#3718) · 406caa5f
      Evan authored
      Previously, passing 0 for newlen would not truncate the string at all.
      This adds handling of this case, freeing the old string and creating a new empty string.
      
      Other changes:
      - Move `src/modules/testmodule.c` to `tests/modules/basics.c`
      - Introduce that basic test into the test suite
      - Add tests to cover StringTruncate
      - Add `test-modules` build target for the main makefile
      - Extend `distclean` build target to clean modules too
      
      (cherry picked from commit 1ccf2ca2)
      406caa5f
    • guybe7's avatar
      Module API for current command name (#8792) · c547dff6
      guybe7 authored
      sometimes you can be very deep in the call stack, without access to argv.
      once you're there you may want your reply/log to contain the command name.
      
      (cherry picked from commit e16d3eb9)
      c547dff6
  6. 03 May, 2021 2 commits
  7. 19 Apr, 2021 1 commit
    • Hanna Fadida's avatar
      Modules: adding a module type for key space notification (#8759) · 53a4d6c3
      Hanna Fadida authored
      Adding a new type mask ​for key space notification, REDISMODULE_NOTIFY_MODULE, to enable unique notifications from commands on REDISMODULE_KEYTYPE_MODULE type keys (which is currently unsupported).
      
      Modules can subscribe to a module key keyspace notification by RM_SubscribeToKeyspaceEvents,
      and clients by notify-keyspace-events of redis.conf or via the CONFIG SET, with the characters 'd' or 'A' 
      (REDISMODULE_NOTIFY_MODULE type mask is part of the '**A**ll' notation for key space notifications).
      
      Refactor: move some pubsub test infra from pubsub.tcl to util.tcl to be re-used by other tests.
      53a4d6c3
  8. 13 Apr, 2021 3 commits
    • Viktor Söderqvist's avatar
      Modules API docs: Sections and links (#8442) · d7920ff9
      Viktor Söderqvist authored
      * Modules API docs: Link API function names to their definitions
      
      Occurrences of API functions are linked to their definition.
      
      A function index with links to all functions is added on the bottom
      of the page.
      
      Comment blocks in module.c starting with a markdown h2 heading are
      used as sections. A table of contents is generated from these
      headings.
      
      The functions names are changed from h2 to h3, since they are now
      rendered as sub-headings within each section.
      
      Existing sections in module.c are used with some minor changes.
      Some documentation text is added or sligtly modified.
      
      The markdown renderer will add IDs which may clash with our
      generated IDs. By prefixing section IDs with "section-" we make
      them different.
      
      Replace double dashes with a unicode long ndash
      d7920ff9
    • Viktor Söderqvist's avatar
      Small doc fix for stream module API (#8757) · 4938052f
      Viktor Söderqvist authored
      In a code example, using RedisModule_FreeString instead of
      RedisModule_Free makes it behave correctly regardless of whether
      automatic memory is used or not.
      4938052f
    • Oran Agra's avatar
      Revert "Fix: server will crash if rdbload or rdbsave method is not provided in... · b278e443
      Oran Agra authored
      Revert "Fix: server will crash if rdbload or rdbsave method is not provided in module (#8670)" (#8771)
      
      This reverts commit 808f3004.
      b278e443
  9. 06 Apr, 2021 1 commit
  10. 01 Apr, 2021 1 commit
  11. 25 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix SLOWLOG for blocked commands (#8632) · 497351ad
      Oran Agra authored
      * SLOWLOG didn't record anything for blocked commands because the client
        was reset and argv was already empty. there was a fix for this issue
        specifically for modules, now it works for all blocked clients.
      * The original command argv (before being re-written) was also reset
        before adding the slowlog on behalf of the blocked command.
      * Latency monitor is now updated regardless of the slowlog flags of the
        command or its execution (their purpose is to hide sensitive info from
        the slowlog, not hide the fact the latency happened).
      * Latency monitor now uses real_cmd rather than c->cmd (which may be
        different if the command got re-written, e.g. GEOADD)
      
      Changes:
      * Unify shared code between slowlog insertion in call() and
        updateStatsOnUnblock(), hopefully prevent future bugs from happening
        due to the later being overlooked.
      * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
        processing.
      * Add a test for SLOWLOG and BLPOP
      
      Notes:
      - real_cmd == c->lastcmd, except inside MULTI and Lua.
      - blocked commands never happen in these cases (MULTI / Lua)
      - real_cmd == c->cmd, except for when the command is rewritten (e.g.
        GEOADD)
      - blocked commands (currently) are never rewritten
      - other than the command's CLIENT_PREVENT_LOGGING, and the
        execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
        avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
        be off when executed from execCommand that runs from an AOF)
      497351ad
  12. 24 Mar, 2021 2 commits
  13. 16 Mar, 2021 1 commit
    • Oran Agra's avatar
      set module eviction context flag only in masters (#8631) · 40d555db
      Oran Agra authored
      REDISMODULE_CTX_FLAGS_EVICT and REDISMODULE_CTX_FLAGS_MAXMEMORY
      shouldn't be set when the module is run inside a replica that doesn't do eviction.
      
      one may argue that the database is under eviction (the master does eviction and sends DELs to the replica).
      but on the other hand, we don't really know the master's configuration.
      all that matters is if the current instance does evictions or not.
      40d555db
  14. 15 Mar, 2021 1 commit
    • guybe7's avatar
      Missing EXEC on modules propagation after failed EVAL execution (#8654) · dba33a94
      guybe7 authored
      1. moduleReplicateMultiIfNeeded should use server.in_eval like
         moduleHandlePropagationAfterCommandCallback
      2. server.in_eval could have been set to 1 and not reset back
         to 0 (a lot of missed early-exits after in_eval is already 1)
      
      Note: The new assertions in processCommand cover (2) and I added
      two module tests to cover (1)
      
      Implications:
      If an EVAL that failed (and thus left server.in_eval=1) runs before a module
      command that replicates, the replication stream will contain MULTI (because
      moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL
      at this point) but not EXEC (because server.in_eval==1)
      This only affects modules as module.c the only user of server.in_eval.
      
      Affects versions 6.2.0, 6.2.1
      dba33a94
  15. 14 Mar, 2021 1 commit
  16. 10 Mar, 2021 2 commits
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
    • guybe7's avatar
      Cleanup ZADD_* flags (#8559) · 61a73de6
      guybe7 authored
      Have a clear separation between in and out flags
      
      Other changes:
      
      delete dead code in RM_ZsetIncrby: if zsetAdd returned error (happens only if
      the result of the operation is NAN or if score is NAN) we return immediately so
      there is no way that zsetAdd succeeded and returned NAN in the out-flags
      61a73de6
  17. 08 Mar, 2021 1 commit
    • guybe7's avatar
      Fix edge-case when a module client is unblocked (#8618) · e58118cd
      guybe7 authored
      Scenario:
      1. A module client is blocked on keys with a timeout
      2. Shortly before the timeout expires, the key is being populated and signaled
         as ready
      3. Redis calls moduleTryServeClientBlockedOnKey (which replies to client) and
         then moduleUnblockClient
      4. moduleUnblockClient doesn't really unblock the client, it writes to
         server.module_blocked_pipe and only marks the BC as unblocked.
      5. beforeSleep kics in, by this time the client still exists and techincally
         timed-out. beforeSleep replies to the timeout client (double reply) and
         only then moduleHandleBlockedClients is called, reading from module_blocked_pipe
         and calling unblockClient
      
      The solution is similar to what was done in moduleTryServeClientBlockedOnKey: we
      should avoid re-processing an already-unblocked client
      e58118cd
  18. 02 Mar, 2021 1 commit
  19. 28 Feb, 2021 2 commits
    • Bonsai's avatar
      Module API for getting user name of a client (#8508) · 17c226b0
      Bonsai authored
      Adding RM_GetClientUserNameById to get the ACL user name of a client connection.
      17c226b0
    • Viktor Söderqvist's avatar
      Shared reusable client for RM_Call() (#8516) · 6122f1c4
      Viktor Söderqvist authored
      A single client pointer is added in the server struct. This is
      initialized by the first RM_Call() and reused for every subsequent
      RM_Call() except if it's already in use, which means that it's not
      used for (recursive) module calls to modules. For these, a new
      "fake" client is created each time.
      
      Other changes:
      * Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels
        when not needed
      6122f1c4
  20. 16 Feb, 2021 1 commit
    • uriyage's avatar
      Adds INFO fields to track fork child progress (#8414) · fd052d2a
      uriyage authored
      * Adding current_save_keys_total and current_save_keys_processed info fields.
        Present in replication, BGSAVE and AOFRW.
      * Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
      * Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
        and module forks.
      fd052d2a
  21. 15 Feb, 2021 2 commits
  22. 10 Feb, 2021 1 commit
  23. 05 Feb, 2021 1 commit
    • Viktor Söderqvist's avatar
      RM_ZsetRem: Delete key if empty (#8453) · aea6e71e
      Viktor Söderqvist authored
      Without this fix, RM_ZsetRem can leave empty sorted sets which are
      not allowed to exist.
      
      Removing from a sorted set while iterating seems to work (while
      inserting causes failed assetions). RM_ZsetRangeEndReached is
      modified to return 1 if the key doesn't exist, to terminate
      iteration when the last element has been removed.
      aea6e71e
  24. 29 Jan, 2021 1 commit
    • filipe oliveira's avatar
      Enabled background and reply time tracking on blocked on keys/blocked on... · f0c5052a
      filipe oliveira authored
      Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
      
      This commit enables tracking time of the background tasks and on replies,
      opening the door for properly tracking commands that rely on blocking / background
       work via the slowlog, latency history, and commandstats. 
      
      Some notes:
      - The time spent blocked waiting for key changes, or blocked on synchronous
        replication is not accounted for. 
      
      - **This commit does not affect latency tracking of commands that are non-blocking
        or do not have background work.** ( meaning that it all stays the same with exception to
        `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
        on background threads ). 
      
      -  Specifically for latency history command we've added a new event class named
        `command-unblocking` that will enable latency monitoring on commands that spawn
        background threads to do the work.
      
      - For blocking commands we're now considering the total time of a command as the
        time spent on call() + the time spent on replying when unblocked.
      
      - For Modules commands that rely on background threads we're now considering the
        total time of a command as the time spent on call (main thread) + the time spent on
        the background thread ( if marked within `RedisModule_MeasureTimeStart()` and
        `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread)
      
      To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on
      a module that blocks the client and sleeps on the background for a given time. 
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout
      - check blocked command with multiple calls RedisModule_MeasureTimeStart()  is tracking the total background time
      - check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
      f0c5052a
  25. 28 Jan, 2021 2 commits
    • guybe7's avatar
      Modules: Add event for fork child birth and termination (#8289) · 01cbf17b
      guybe7 authored
      Useful to avoid doing background jobs that can cause excessive COW
      01cbf17b
    • Viktor Söderqvist's avatar
      Add modules API for streams (#8288) · 4355145a
      Viktor Söderqvist authored
      APIs added for these stream operations: add, delete, iterate and
      trim (by ID or maxlength). The functions are prefixed by RM_Stream.
      
      * RM_StreamAdd
      * RM_StreamDelete
      * RM_StreamIteratorStart
      * RM_StreamIteratorStop
      * RM_StreamIteratorNextID
      * RM_StreamIteratorNextField
      * RM_StreamIteratorDelete
      * RM_StreamTrimByLength
      * RM_StreamTrimByID
      
      The type RedisModuleStreamID is added and functions for converting
      from and to RedisModuleString.
      
      * RM_CreateStringFromStreamID
      * RM_StringToStreamID
      
      Whenever the stream functions return REDISMODULE_ERR, errno is set to
      provide additional error information.
      
      Refactoring: The zset iterator fields in the RedisModuleKey struct
      are wrapped in a union, to allow the same space to be used for type-
      specific info for streams and allow future use for other key types.
      4355145a
  26. 20 Jan, 2021 1 commit
    • Viktor Söderqvist's avatar
      More modules API ref formatting fixes (#8344) · 16258f21
      Viktor Söderqvist authored
      Fix broken formatting in `RM_Call` and `RM_CreateDataType`,
      `RM_SubscribeToServerEvent` (nested lists, etc. in list items).
      
      Unhide docs of `RM_LoadDataTypeFromString` and
      `RM_SaveDataTypeToString` by removing blank line between docs and
      function.
      
      Clarification added to `RM__Assert`: Recommentation to use the
      `RedisModule_Assert` macro instead.
      
      All names containing underscores (variable and macro names) are
      wrapped in backticks (if not already wrapped in backticks). This
      prevents underscore from being interpreted as italics in some
      cases.
      
      Names including a wildcard star, e.g. RM_Defrag*(), is wrapped in
      backticks (and RM replaced by RedisModule in this case). This
      prevents the * from being interpreted as an italics marker.
      
      A list item with a sublist, a paragraph and another sublist is a
      combination which seems impossible to achieve with RedCarped
      markdown, so the one occurrence of this is rewritten.
      
      Various trivial changes (typos, backticks, etc.).
      
      Ruby script:
      
      * Replace `RM_Xyz` with `RedisModule_Xyz` in docs. (RM is correct
        when refering to the C code but RedisModule is correct in the
        API docs.)
      * Automatic backquotes around C functions like `malloc()`.
      * Turn URLs into links. The link text is the URL itself.
      * Don't add backticks inside bold (**...**)
      16258f21