1. 21 Nov, 2024 4 commits
    • Ozan Tezcan's avatar
      Fix memory leak of jemalloc tcache on function flush command (#13661) · 9ebf80a2
      Ozan Tezcan authored
      Starting from https://github.com/redis/redis/pull/13133
      
      , we allocate a
      jemalloc thread cache and use it for lua vm.
      On certain cases, like `script flush` or `function flush` command, we
      free the existing thread cache and create a new one.
      
      Though, for `function flush`, we were not actually destroying the
      existing thread cache itself. Each call creates a new thread cache on
      jemalloc and we leak the previous thread cache instances. Jemalloc
      allows maximum 4096 thread cache instances. If we reach this limit,
      Redis prints "Failed creating the lua jemalloc tcache" log and abort.
      
      There are other cases that can cause this memory leak, including
      replication scenarios when emptyData() is called.
      
      The implication is that it looks like redis `used_memory` is low, but
      `allocator_allocated` and RSS remain high.
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      9ebf80a2
    • Moti Cohen's avatar
      modules API: Support register unprefixed config parameters (#13656) · 15563450
      Moti Cohen authored
      PR #10285 introduced support for modules to register four types of
      configurations — Bool, Numeric, String, and Enum. Accessible through the
      Redis config file and the CONFIG command.
      
      With this PR, it will be possible to register configuration parameters
      without automatically prefixing the parameter names. This provides
      greater flexibility in configuration naming, enabling, for instance,
      both `bf-initial-size` or `initial-size` to be defined in the module
      without automatically prefixing with `<MODULE-NAME>.`. In addition it
      will also be possible to create a single additional alias via the same
      API. This brings us another step closer to integrate modules into redis
      core.
      
      **Example:** Register a configuration parameter `bf-initial-size` with
      an alias `initial-size` without the automatic module name prefix, set
      with new `REDISMODULE_CONFIG_UNPREFIXED` flag:
      ```
      RedisModule_RegisterBoolConfig(ctx, "bf-initial-size|initial-size", default_val, optflags | REDISMODULE_CONFIG_UNPREFIXED, getfn, setfn, applyfn, privdata);
      ```
      # API changes
      Related functions that now support unprefixed configuration flag
      (`REDISMODULE_CONFIG_UNPREFIXED`) along with optional alias:
      ```
      RedisModule_RegisterBoolConfig
      RedisModule_RegisterEnumConfig
      RedisModule_RegisterNumericConfig
      RedisModule_RegisterStringConfig
      ```
      
      # Implementation Details:
      `config.c`: On load server configuration, at function
      `loadServerConfigFromString()`, it collects all unknown configurations
      into `module_configs_queue` dictionary. These may include valid module
      configurations or invalid ones. They will be validated later by
      `loadModuleConfigs()` against the configurations declared by the loaded
      module(s).
      `Module.c:` The `ModuleConfig` structure has been modified to store now:
      (1) Full configuration name (2) Alias (3) Unprefixed flag status -
      ensuring that configurations retain their original registration format
      when triggered in notifications.
      
      Added error printout:
      This change introduces an error printout for unresolved configurations,
      detailing each unresolved parameter detected during startup. The last
      line in the output existed prior to this change and has been retained to
      systems relies on it:
      ```
      595011:M 18 Nov 2024 08:26:23.616 # Unresolved Configuration(s) Detected:
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'bf-initiel-size 8'
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'search-sizex 32'
      595011:M 18 Nov 2024 08:26:23.616 # Module Configuration detected without loadmodule directive or no ApplyConfig call: aborting
      ```
      
      # Backward Compatibility:
      Existing modules will function without modification, as the new
      functionality only applies if REDISMODULE_CONFIG_UNPREFIXED is
      explicitly set.
      
      # Module vs. Core API Conflict Behavior
      The new API allows to modules loading duplication of same configuration
      name or same configuration alias, just like redis core configuration
      allows (i.e. the users sets two configs with a different value, but
      these two configs are actually the same one). Unlike redis core, given a
      name and its alias, it doesn't allow have both configuration on load. To
      implement it, it is required to modify DS `module_configs_queue` to
      reflect the order of their loading and later on, during
      `loadModuleConfigs()`, resolve pairs of names and aliases and which one
      is the last one to apply. "Relaxing" this limitation can be deferred to
      a future update if necessary, but for now, we error in this case.
      15563450
    • Oran Agra's avatar
      Add Lua VM memory to memory overhead, now that it's part of zmalloc (#13660) · 79fd2558
      Oran Agra authored
      To complement the work done in #13133.
      it added the script VMs memory to be counted as part of zmalloc, but
      that means they
      should be also counted as part of the non-value overhead.
      
      this commit contains some refactoring to make variable names and
      function names less confusing.
      it also adds a new field named `script.VMs` into the `MEMORY STATS`
      command.
      
      additionally, clear scripts and stats between tests in external mode
      (which is related to how this issue was discovered)
      79fd2558
    • nafraf's avatar
      Fix module loadex command crash due to invalid config (#13653) · 5b84dc96
      nafraf authored
      Fix to https://github.com/redis/redis/issues/13650
      
      
      
      providing an invalid config to a module with datatype crashes when redis
      tries to unload the module due to the invalid config
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      5b84dc96
  2. 11 Nov, 2024 1 commit
    • Ozan Tezcan's avatar
      Print command tokens on a crash when hide-user-data-from-log is enabled (#13639) · 54038811
      Ozan Tezcan authored
      If `hide-user-data-from-log` config is enabled, we don't print client
      argv in the crashlog to avoid leaking user info.
      Though, debugging a crash becomes harder as we don't see the command
      arguments causing the crash.
      
      With this PR, we'll be printing command tokens to the log. As we have
      command tokens defined in json schema for each command, using this data,
      we can find tokens in the client argv.
      
      e.g. 
      `SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the
      log.
      
      Modules should introduce their command structure via
      `RM_SetCommandInfo()`.
      Then, on a crash we'll able to know module command tokens.
      54038811
  3. 30 Oct, 2024 1 commit
  4. 29 Oct, 2024 1 commit
    • Moti Cohen's avatar
      Add KEYSIZES section to INFO (#13592) · 2ec78d26
      Moti Cohen authored
      This PR adds a new section to the `INFO` command output, called
      `keysizes`. This section provides detailed statistics on the
      distribution of key sizes for each data type (strings, lists, sets,
      hashes and zsets) within the dataset. The distribution is tracked using
      a base-2 logarithmic histogram.
      
      # Motivation
      Currently, Redis lacks a built-in feature to track key sizes and item
      sizes per data type at a granular level. Understanding the distribution
      of key sizes is critical for monitoring memory usage and optimizing
      performance, particularly in large datasets. This enhancement will allow
      users to inspect the size distribution of keys directly from the `INFO`
      command, assisting with performance analysis and capacity planning.
      
      # Changes
      New Section in `INFO` Command: A new section called `keysizes` has been
      added to the `INFO` command output. This section reports a per-database,
      per-type histogram of key sizes. It provides insights into how many keys
      fall into specific size ranges (represented in powers of 2).
      
      **Example output:**
      ```
      127.0.0.1:6379> INFO keysizes
      # Keysizes
      db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
      db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
      db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
      db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
      ```
      ## Future Use Cases:
      The key size distribution is collected per slot as well, laying the
      groundwork for future enhancements related to Redis Cluster.
      2ec78d26
  5. 22 Oct, 2024 1 commit
  6. 17 Oct, 2024 1 commit
    • Yuan Wang's avatar
      Clean up .rediscli_history_test temporary file (#13601) · b71a610f
      Yuan Wang authored
      After running test in local, there will be a file named
      `.rediscli_history_test`, and it is not in `.gitignore` file, so this is
      considered to have changed the code base. It is a little annoying, this
      commit just clean up the temporary file.
      
      We should delete `.rediscli_history_test` in the end since the second
      server tests also write somethings into it, to make it corresponding, i
      put `set ::env(REDISCLI_HISTFILE) ".rediscli_history_test"` at the
      beginning.
      
      Maybe we also can add this file into `.gitignore`
      b71a610f
  7. 15 Oct, 2024 1 commit
  8. 12 Oct, 2024 1 commit
  9. 08 Oct, 2024 3 commits
  10. 29 Sep, 2024 1 commit
    • Moti Cohen's avatar
      Add new SFLUSH command to cluster for slot-based FLUSH (#13564) · d092d64d
      Moti Cohen authored
      This PR introduces a new `SFLUSH` command to cluster mode that allows
      partial flushing of nodes based on specified slot ranges. Current
      implementation is designed to flush all slots of a shard, but future
      extensions could allow for more granular flushing.
      
      **Command Usage:**
      `SFLUSH <start-slot> <end-slot> [<start-slot> <end-slot>]* [SYNC|ASYNC]`
      
      This command removes all data from the specified slots, either
      synchronously or asynchronously depending on the optional SYNC/ASYNC
      argument.
      
      **Functionality:**
      Current imp of `SFLUSH` command verifies that the provided slot ranges
      are valid and cover all of the node's slots before proceeding. If slots
      are partially or incorrectly specified, the command will fail and return
      an error, ensuring that all slots of a node must be fully covered for
      the flush to proceed.
      
      The function supports both synchronous (default) and asynchronous
      flushing. In addition, if possible, SFLUSH SYNC will be run as blocking
      ASYNC as an optimization.
      d092d64d
  11. 23 Sep, 2024 2 commits
  12. 19 Sep, 2024 1 commit
    • Moti Cohen's avatar
      Extend modules API to read also expired keys and subkeys (#13526) · 3a3cacfe
      Moti Cohen authored
      The PR extends `RedisModule_OpenKey`'s flags to include
      `REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired
      keys.
      
      It also allows to access expired subkeys. Currently relevant only for
      hash fields
      and has its impact on `RM_HashGet` and `RM_Scan`.
      3a3cacfe
  13. 13 Sep, 2024 1 commit
  14. 12 Sep, 2024 3 commits
    • Filipe Oliveira (Redis)'s avatar
      Optimize SSCAN command in case of listpack or intset encoding: avoid the usage... · f2f85ba3
      Filipe Oliveira (Redis) authored
      Optimize SSCAN command in case of listpack or intset encoding: avoid the usage of intermediate list. From 2N to N iterations (#13530)
      
      On SSCAN, in case of listpack and intset encoding we actually reply the
      entire set, and always reply with the cursor 0.
      
      For those cases, we don't need to accumulate the replies in a list and
      can completely avoid the overhead of list appending and then iterating
      over the list again -- meaning we do N iterations instead of 2N
      iterations over the SET and save intermediate memory as well.
      
      Preliminary benchmarks, `SSCAN set:100 0`, showcased an improvement of
      60% as visible bellow on a SET with 100 string elements (listpack
      encoded).
      f2f85ba3
    • Moti Cohen's avatar
      HFE - Fix key ref by the hash on RENAME/MOVE/SWAPDB/RESTORE (#13539) · 9a89e32a
      Moti Cohen authored
      If the hash previously had HFEs (hash-fields with expiration) but later no longer
      does, the key ref in the hash might become outdated after a MOVE, COPY,
      RENAME or RESTORE operation. These commands maintain the key ref only
      if HFEs are present. That is, we can only be sure that key ref is valid as long as the
      hash has HFEs.
      9a89e32a
    • Oran Agra's avatar
      RED-129256, Fix TOUCH command from script in no-touch mode (#13512) · 610eb26c
      Oran Agra authored
      
      
      When a client in no-touch mode issues a TOUCH command on a key, the
      key’s access time should be updated, but in scripts, and module's
      RM_Call, it isn’t updated.
      Command proc should be matched to the executing client, not the current
      client.
      Co-authored-by: default avatarUdi Ron <udi@speedb.io>
      610eb26c
  15. 08 Sep, 2024 1 commit
    • Ozan Tezcan's avatar
      Fix flaky replication tests (#13518) · ac03e372
      Ozan Tezcan authored
      #13495 introduced a change to reply -LOADING while flushing existing db on a replica. Some of our tests are
       sensitive to this change and do no expect -LOADING reply.
      
      Fixing a couple of tests that fail time to time.
      ac03e372
  16. 05 Sep, 2024 1 commit
  17. 04 Sep, 2024 3 commits
  18. 03 Sep, 2024 2 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Added new defrag API to allocate and free raw memory. (#13509) · d3d94ccf
      Meir Shpilraien (Spielrein) authored
      All the defrag allocations API expects to get a value and replace it, leaving the old value untouchable.
      In some cases a value might be shared between multiple keys, in such cases we can not simply replace
      it when the defrag callback is called.
      
      To allow support such use cases, the PR adds two new API's to the defrag API:
      
      1. `RM_DefragAllocRaw` - allocate memory base on a given size.
      2. `RM_DefragFreeRaw` - Free the given pointer.
      
      Those API's avoid using tcache so they operate just like `RM_DefragAlloc` but allows the user to split
      the allocation and the memory free operations into two stages and control when those happen.
      
      In addition the PR adds new API to allow the module to receive notifications when defrag start and end: `RM_RegisterDefragCallbacks`
      Those callbacks are the same as `RM_RegisterDefragFunc` but promised to be called and the start
      and the end of the defrag process.
      d3d94ccf
    • Ozan Tezcan's avatar
      Reply LOADING on replica while flushing the db (#13495) · a7afd1d2
      Ozan Tezcan authored
      On a full sync, replica starts discarding existing db. If the existing 
      db is huge and flush is happening synchronously, replica may become 
      unresponsive. 
      
      Adding a change to yield back to event loop while flushing db on 
      a replica. Replica will reply -LOADING in this case. Note that while 
      replica is loading the new rdb, it may get an error and start flushing
      the partial db. This step may take a long time as well. Similarly, 
      replica will reply -LOADING in this case. 
      
      To call processEventsWhileBlocked() and reply -LOADING, we need to do:
      - Set connSetReadHandler() null not to process further data from the master
      - Set server.loading flag
      - Call blockingOperationStarts()
      
      rdbload() already does these steps and calls processEventsWhileBlocked()
      while loading the rdb. Added a new call rdbLoadWithEmptyFunc() which 
      accepts callback to flush db before loading rdb or when an error 
      happens while loading. 
      
      For diskless replication, doing something similar and calling emptyData()
      after setting required flags.
      
      Additional changes:
      - Allow `appendonly` config change during loading. 
       Config can be changed while loading data on startup or on replication 
       when slave is loading RDB. We allow config change command to update 
       `server.aof_enabled` and then lazily apply config change after loading
       operation is completed.
       
       - Added a test for `replica-lazy-flush` config
      a7afd1d2
  19. 29 Aug, 2024 1 commit
    • Oran Agra's avatar
      testsuite --dump-logs works on servers started before the test (#13500) · 3fcddfb6
      Oran Agra authored
      so far ./runtest --dump-logs used work for servers started within the
      test proc.
      now it'll also work on servers started outside the test proc scope.
      the downside is that these logs can be huge if they served many tests
      and not just the failing one.
      but for some rare failures, we rather have that than nothing.
      this feature isn't enabled y default, but is used by our GH actions.
      3fcddfb6
  20. 20 Aug, 2024 2 commits
    • Zihao Lin's avatar
      Improve GETRANGE command behavior (#12272) · 6ceadfb5
      Zihao Lin authored
      
      
      Fixed the issue about GETRANGE and SUBSTR command
      return unexpected result caused by the `start` and `end` out of
      definition range of string.
      
      ---
      ## break change
      Before this PR, when negative `end` was out of range (i.e., end <
      -strlen), we would fix it to 0 to get the substring, which also resulted
      in the first character still being returned for this kind of out of
      range.
      After this PR, we ensure that `GETRANGE` returns an empty bulk when the
      negative end index is out of range.
      
      Closes #11738
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      6ceadfb5
    • judeng's avatar
      improve performance for scan command when matching data type (#12395) · 7f0a7f0a
      judeng authored
      Move the TYPE filtering to the scan callback so that avoided the
      `lookupKey` operation. This is the follow-up to #12209 . In this thread
      we introduced two breaking changes:
      1. we will not attempt to do lazy expire (delete) a key that was
      filtered by not matching the TYPE (like we already do for MATCH
      pattern).
      2. when the specified key TYPE filter is an unknown type, server will
      reply a error immediately instead of doing a full scan that comes back
      empty handed.
      7f0a7f0a
  21. 16 Aug, 2024 1 commit
    • debing.sun's avatar
      Fix incorrect lag due to trimming stream via XTRIM command (#13473) · 2b88db90
      debing.sun authored
      ## Describe
      When using the `XTRIM` command to trim a stream, it does not update the
      maximal tombstone (`max_deleted_entry_id`). This leads to an issue where
      the lag calculation incorrectly assumes that there are no tombstones
      after the consumer group's last_id, resulting in an inaccurate lag.
      
      The reason XTRIM doesn't need to update the maximal tombstone is that it
      always trims from the beginning of the stream. This means that it
      consistently changes the position of the first entry, leading to the
      following scenarios:
      
      1) First entry trimmed after maximal tombstone:
      If the first entry is trimmed to a position after the maximal tombstone,
      all tombstones will be before the first entry, so they won't affect the
      consumer group's lag.
      
      2) First entry trimmed before maximal tombstone:
      If the first entry is trimmed to a position before the maximal
      tombstone, the maximal tombstone will not be updated.
      
      ## Solution
      Therefore, this PR optimizes the lag calculation by ensuring that when
      both the consumer group's last_id and the maximal tombstone are behind
      the first entry, the consumer group's lag is always equal to the number
      of remaining elements in the stream.
      
      Supplement to PR https://github.com/redis/redis/pull/13338
      2b88db90
  22. 14 Aug, 2024 1 commit
  23. 11 Aug, 2024 1 commit
    • Moti Cohen's avatar
      On HDEL last field with expiry, update global HFE DS (#13470) · 806459f4
      Moti Cohen authored
      Hash field expiration is optimized to avoid frequent update global HFE DS for
      each field deletion. Eventually active-expiration will run and update or remove
      the hash from global HFE DS gracefully. Nevertheless, statistic "subexpiry"
      might reflect wrong number of hashes with HFE to the user if HDEL deletes
      the last field with expiration in hash (yet there are more fields without expiration).
      
      Following this change, if HDEL the last field with expiration in the hash then
      take care to remove the hash from global HFE DS as well.
      806459f4
  24. 08 Aug, 2024 1 commit
  25. 03 Aug, 2024 1 commit
  26. 01 Aug, 2024 1 commit
  27. 31 Jul, 2024 1 commit
  28. 30 Jul, 2024 1 commit