1. 21 Nov, 2024 4 commits
    • Ozan Tezcan's avatar
      Fix memory leak of jemalloc tcache on function flush command (#13661) · 9ebf80a2
      Ozan Tezcan authored
      Starting from https://github.com/redis/redis/pull/13133
      
      , we allocate a
      jemalloc thread cache and use it for lua vm.
      On certain cases, like `script flush` or `function flush` command, we
      free the existing thread cache and create a new one.
      
      Though, for `function flush`, we were not actually destroying the
      existing thread cache itself. Each call creates a new thread cache on
      jemalloc and we leak the previous thread cache instances. Jemalloc
      allows maximum 4096 thread cache instances. If we reach this limit,
      Redis prints "Failed creating the lua jemalloc tcache" log and abort.
      
      There are other cases that can cause this memory leak, including
      replication scenarios when emptyData() is called.
      
      The implication is that it looks like redis `used_memory` is low, but
      `allocator_allocated` and RSS remain high.
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      9ebf80a2
    • Moti Cohen's avatar
      modules API: Support register unprefixed config parameters (#13656) · 15563450
      Moti Cohen authored
      PR #10285 introduced support for modules to register four types of
      configurations — Bool, Numeric, String, and Enum. Accessible through the
      Redis config file and the CONFIG command.
      
      With this PR, it will be possible to register configuration parameters
      without automatically prefixing the parameter names. This provides
      greater flexibility in configuration naming, enabling, for instance,
      both `bf-initial-size` or `initial-size` to be defined in the module
      without automatically prefixing with `<MODULE-NAME>.`. In addition it
      will also be possible to create a single additional alias via the same
      API. This brings us another step closer to integrate modules into redis
      core.
      
      **Example:** Register a configuration parameter `bf-initial-size` with
      an alias `initial-size` without the automatic module name prefix, set
      with new `REDISMODULE_CONFIG_UNPREFIXED` flag:
      ```
      RedisModule_RegisterBoolConfig(ctx, "bf-initial-size|initial-size", default_val, optflags | REDISMODULE_CONFIG_UNPREFIXED, getfn, setfn, applyfn, privdata);
      ```
      # API changes
      Related functions that now support unprefixed configuration flag
      (`REDISMODULE_CONFIG_UNPREFIXED`) along with optional alias:
      ```
      RedisModule_RegisterBoolConfig
      RedisModule_RegisterEnumConfig
      RedisModule_RegisterNumericConfig
      RedisModule_RegisterStringConfig
      ```
      
      # Implementation Details:
      `config.c`: On load server configuration, at function
      `loadServerConfigFromString()`, it collects all unknown configurations
      into `module_configs_queue` dictionary. These may include valid module
      configurations or invalid ones. They will be validated later by
      `loadModuleConfigs()` against the configurations declared by the loaded
      module(s).
      `Module.c:` The `ModuleConfig` structure has been modified to store now:
      (1) Full configuration name (2) Alias (3) Unprefixed flag status -
      ensuring that configurations retain their original registration format
      when triggered in notifications.
      
      Added error printout:
      This change introduces an error printout for unresolved configurations,
      detailing each unresolved parameter detected during startup. The last
      line in the output existed prior to this change and has been retained to
      systems relies on it:
      ```
      595011:M 18 Nov 2024 08:26:23.616 # Unresolved Configuration(s) Detected:
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'bf-initiel-size 8'
      595011:M 18 Nov 2024 08:26:23.616 #  >>> 'search-sizex 32'
      595011:M 18 Nov 2024 08:26:23.616 # Module Configuration detected without loadmodule directive or no ApplyConfig call: aborting
      ```
      
      # Backward Compatibility:
      Existing modules will function without modification, as the new
      functionality only applies if REDISMODULE_CONFIG_UNPREFIXED is
      explicitly set.
      
      # Module vs. Core API Conflict Behavior
      The new API allows to modules loading duplication of same configuration
      name or same configuration alias, just like redis core configuration
      allows (i.e. the users sets two configs with a different value, but
      these two configs are actually the same one). Unlike redis core, given a
      name and its alias, it doesn't allow have both configuration on load. To
      implement it, it is required to modify DS `module_configs_queue` to
      reflect the order of their loading and later on, during
      `loadModuleConfigs()`, resolve pairs of names and aliases and which one
      is the last one to apply. "Relaxing" this limitation can be deferred to
      a future update if necessary, but for now, we error in this case.
      15563450
    • Oran Agra's avatar
      Add Lua VM memory to memory overhead, now that it's part of zmalloc (#13660) · 79fd2558
      Oran Agra authored
      To complement the work done in #13133.
      it added the script VMs memory to be counted as part of zmalloc, but
      that means they
      should be also counted as part of the non-value overhead.
      
      this commit contains some refactoring to make variable names and
      function names less confusing.
      it also adds a new field named `script.VMs` into the `MEMORY STATS`
      command.
      
      additionally, clear scripts and stats between tests in external mode
      (which is related to how this issue was discovered)
      79fd2558
    • nafraf's avatar
      Fix module loadex command crash due to invalid config (#13653) · 5b84dc96
      nafraf authored
      Fix to https://github.com/redis/redis/issues/13650
      
      
      
      providing an invalid config to a module with datatype crashes when redis
      tries to unload the module due to the invalid config
      
      ---------
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      5b84dc96
  2. 14 Nov, 2024 1 commit
  3. 12 Nov, 2024 1 commit
  4. 11 Nov, 2024 1 commit
    • Ozan Tezcan's avatar
      Print command tokens on a crash when hide-user-data-from-log is enabled (#13639) · 54038811
      Ozan Tezcan authored
      If `hide-user-data-from-log` config is enabled, we don't print client
      argv in the crashlog to avoid leaking user info.
      Though, debugging a crash becomes harder as we don't see the command
      arguments causing the crash.
      
      With this PR, we'll be printing command tokens to the log. As we have
      command tokens defined in json schema for each command, using this data,
      we can find tokens in the client argv.
      
      e.g. 
      `SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the
      log.
      
      Modules should introduce their command structure via
      `RM_SetCommandInfo()`.
      Then, on a crash we'll able to know module command tokens.
      54038811
  5. 08 Nov, 2024 1 commit
    • Nugine's avatar
      Optimize PFCOUNT, PFMERGE command by SIMD acceleration (#13558) · fdeb9762
      Nugine authored
      This PR optimizes the performance of HyperLogLog commands (PFCOUNT,
      PFMERGE) by adding AVX2 fast paths.
      
      Two AVX2 functions are added for conversion between raw representation
      and dense representation. They are 15 ~ 30 times faster than scalar
      implementaion. Note that sparse representation is not accelerated.
      
      AVX2 fast paths are enabled when the CPU supports AVX2 (checked at
      runtime) and the hyperloglog configuration is default (HLL_REGISTERS ==
      16384 && HLL_BITS == 6).
      
      When merging 3 dense hll structures, the benchmark shows a 12x speedup
      compared to the scalar version.
      
      ```
      pfcount key1 key2 key3
      pfmerge keyall key1 key2 key3
      ```
      
      ```
      ======================================================================================================
      Type             Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
      ------------------------------------------------------------------------------------------------------
      PFCOUNT-scalar    5570.09        35.89060        32.51100        65.27900        69.11900       299.17
      PFCOUNT-avx2     72604.92         2.82072         2.73500         5.50300         7.13500      3899.68
      ------------------------------------------------------------------------------------------------------
      PFMERGE-scalar    7879.13        25.52156        24.19100        46.33500        48.38300       492.45
      PFMERGE-avx2    126448.64         1.58120         1.53500         3.08700         4.89500      7903.04
      ------------------------------------------------------------------------------------------------------
      
      scalar: redis:unstable   9906daf5
      avx2:   Nugine:hll-simd  02e09f85ac07eace50ebdddd0fd70822f7b9152d 
      
      CPU:    13th Gen Intel® Core™ i9-13900H × 20
      Memory: 32.0 GiB
      OS:     Ubuntu 22.04.5 LTS
      ```
      
      Experiment repo: https://github.com/Nugine/redis-hyperloglog
      Benchmark script:
      https://github.com/Nugine/redis-hyperloglog/blob/main/scripts/memtier.sh
      Algorithm:
      https://github.com/Nugine/redis-hyperloglog/blob/main/cpp/bench.cpp
      
      
      
      resolves #13551
      
      ---------
      Co-authored-by: default avatarYuan Wang <wangyuancode@163.com>
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      fdeb9762
  6. 04 Nov, 2024 1 commit
    • David Dougherty's avatar
      Update old links for modules-api-ref.md (#13479) · 9906daf5
      David Dougherty authored
      This PR replaces old .../topics/... links with current links,
      specifically for the modules-api-ref.md file and the new automation that
      Paolo Lazzari is working on. A few of the topics links have redirects,
      but some don't. Best to use updated links.
      9906daf5
  7. 30 Oct, 2024 3 commits
  8. 29 Oct, 2024 1 commit
    • Moti Cohen's avatar
      Add KEYSIZES section to INFO (#13592) · 2ec78d26
      Moti Cohen authored
      This PR adds a new section to the `INFO` command output, called
      `keysizes`. This section provides detailed statistics on the
      distribution of key sizes for each data type (strings, lists, sets,
      hashes and zsets) within the dataset. The distribution is tracked using
      a base-2 logarithmic histogram.
      
      # Motivation
      Currently, Redis lacks a built-in feature to track key sizes and item
      sizes per data type at a granular level. Understanding the distribution
      of key sizes is critical for monitoring memory usage and optimizing
      performance, particularly in large datasets. This enhancement will allow
      users to inspect the size distribution of keys directly from the `INFO`
      command, assisting with performance analysis and capacity planning.
      
      # Changes
      New Section in `INFO` Command: A new section called `keysizes` has been
      added to the `INFO` command output. This section reports a per-database,
      per-type histogram of key sizes. It provides insights into how many keys
      fall into specific size ranges (represented in powers of 2).
      
      **Example output:**
      ```
      127.0.0.1:6379> INFO keysizes
      # Keysizes
      db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
      db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
      db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
      db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
      ```
      ## Future Use Cases:
      The key size distribution is collected per slot as well, laying the
      groundwork for future enhancements related to Redis Cluster.
      2ec78d26
  9. 28 Oct, 2024 1 commit
    • Shockingly Good's avatar
      Fix crash in RM_GetCurrentUserName() when the user isn't accessible (#13619) · 611c9502
      Shockingly Good authored
      The crash happens whenever the user isn't accessible, for example, it
      isn't set for the context (when it is temporary) or in some other cases
      like `notifyKeyspaceEvent`. To properly check for the ACL compliance, we
      need to get the user name and the user to invoke other APIs. However, it
      is not possible if it crashes, and it is impossible to work that around
      in the code since we don't know (and **shouldn't know**!) when it is
      available and when it is not.
      611c9502
  10. 22 Oct, 2024 1 commit
  11. 18 Oct, 2024 1 commit
  12. 17 Oct, 2024 2 commits
    • hanhui365's avatar
      Optimize bitcount command by using popcnt (#13359) · 3788a055
      hanhui365 authored
      
      
      Nowadays popcnt instruction is almost supported by X86 machine, which is
      used to calculate "Hamming weight", it can bring much performance boost
      in redis bitcount comand.
      
      ---------
      
      Signed-off-by: hanhui365(hanhui@hygon.cn)
      Co-authored-by: default avatardebing.sun <debing.sun@redis.com>
      Co-authored-by: default avataroranagra <oran@redislabs.com>
      Co-authored-by: default avatarNugine <nugine@foxmail.com>
      3788a055
    • Yuan Wang's avatar
      Clean up .rediscli_history_test temporary file (#13601) · b71a610f
      Yuan Wang authored
      After running test in local, there will be a file named
      `.rediscli_history_test`, and it is not in `.gitignore` file, so this is
      considered to have changed the code base. It is a little annoying, this
      commit just clean up the temporary file.
      
      We should delete `.rediscli_history_test` in the end since the second
      server tests also write somethings into it, to make it corresponding, i
      put `set ::env(REDISCLI_HISTFILE) ".rediscli_history_test"` at the
      beginning.
      
      Maybe we also can add this file into `.gitignore`
      b71a610f
  13. 15 Oct, 2024 3 commits
  14. 12 Oct, 2024 1 commit
  15. 10 Oct, 2024 1 commit
    • guybe7's avatar
      Cleanups related to expiry/eviction (#13591) · a38c29b6
      guybe7 authored
      1. `dbRandomKey`: excessive call to `dbFindExpires` (will always return
      1 if `allvolatile` + anyway called inside `expireIfNeeded`
      2. Add `deleteKeyAndPropagate` that is used by both expiry/eviction
      3. Change the order of calls in `expireIfNeeded` to save redundant calls
      to `keyIsExpired`
      4. `expireIfNeeded`: move `OBJ_STATIC_REFCOUNT` to
      `deleteKeyAndPropagate`
      5. `performEvictions` now uses `deleteEvictedKeyAndPropagate`
      6. active-expire: moved `postExecutionUnitOperations` inside
      `activeExpireCycleTryExpire`
      7. `activeExpireCycleTryExpire`: less indentation + expire a key if `now
      == t`
      8. rename `lazy_expire_disabled` to `allow_access_expired`
      a38c29b6
  16. 08 Oct, 2024 5 commits
  17. 29 Sep, 2024 1 commit
    • Moti Cohen's avatar
      Add new SFLUSH command to cluster for slot-based FLUSH (#13564) · d092d64d
      Moti Cohen authored
      This PR introduces a new `SFLUSH` command to cluster mode that allows
      partial flushing of nodes based on specified slot ranges. Current
      implementation is designed to flush all slots of a shard, but future
      extensions could allow for more granular flushing.
      
      **Command Usage:**
      `SFLUSH <start-slot> <end-slot> [<start-slot> <end-slot>]* [SYNC|ASYNC]`
      
      This command removes all data from the specified slots, either
      synchronously or asynchronously depending on the optional SYNC/ASYNC
      argument.
      
      **Functionality:**
      Current imp of `SFLUSH` command verifies that the provided slot ranges
      are valid and cover all of the node's slots before proceeding. If slots
      are partially or incorrectly specified, the command will fail and return
      an error, ensuring that all slots of a node must be fully covered for
      the flush to proceed.
      
      The function supports both synchronous (default) and asynchronous
      flushing. In addition, if possible, SFLUSH SYNC will be run as blocking
      ASYNC as an optimization.
      d092d64d
  18. 25 Sep, 2024 2 commits
  19. 23 Sep, 2024 2 commits
  20. 19 Sep, 2024 1 commit
    • Moti Cohen's avatar
      Extend modules API to read also expired keys and subkeys (#13526) · 3a3cacfe
      Moti Cohen authored
      The PR extends `RedisModule_OpenKey`'s flags to include
      `REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired
      keys.
      
      It also allows to access expired subkeys. Currently relevant only for
      hash fields
      and has its impact on `RM_HashGet` and `RM_Scan`.
      3a3cacfe
  21. 18 Sep, 2024 1 commit
  22. 15 Sep, 2024 3 commits
  23. 13 Sep, 2024 2 commits