1. 03 May, 2024 1 commit
  2. 18 Apr, 2024 1 commit
    • Moti Cohen's avatar
      Hash Field Expiration - Basic support · c18ff056
      Moti Cohen authored
      - Add ebuckets & mstr data structures
      - Integrate active & lazy expiration
      - Add most of the commands 
      - Add support for dict (listpack is missing)
      TODOs:  RDB, notification, listpack, HSET, HGETF, defrag, aof
      c18ff056
  3. 01 Mar, 2024 1 commit
    • Chen Tianjie's avatar
      Add overhead of all DBs and rehashing dict count to info. (#12913) · 4cae99e7
      Chen Tianjie authored
      
      
      Sometimes we need to make fast judgement about why Redis is suddenly
      taking more memory. One of the reasons is main DB's dicts doing
      rehashing.
      
      We may use `MEMORY STATS` to monitor the overhead memory of each DB, but
      there still lacks a total sum to show an overall trend. So this PR adds
      the total overhead of all DBs to `INFO MEMORY` section, together with
      the total count of rehashing DB dicts, providing some intuitive metrics
      about main dicts rehashing.
      
      This PR adds the following metrics to INFO MEMORY
      * `mem_overhead_db_hashtable_rehashing` - only size of ht[0] in
      dictionaries we're rehashing (i.e. the memory that's gonna get released
      soon)
      
      and a similar ones to MEMORY STATS:
      * `overhead.db.hashtable.lut` (complements the existing
      `overhead.hashtable.main` and `overhead.hashtable.expires` which also
      counts the `dictEntry` structs too)
      * `overhead.db.hashtable.rehashing` - temporary rehashing overhead.
      * `db.dict.rehashing.count` - number of top level dictionaries being
      rehashed.
      
      ---------
      Co-authored-by: default avatarzhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      4cae99e7
  4. 29 Feb, 2024 1 commit
    • Binbin's avatar
      Fix propagation of entries_read by calling streamPropagateGroupID unconditionally (#12898) · f17381a3
      Binbin authored
      In XREADGROUP ACK, because streamPropagateXCLAIM does not propagate
      entries-read, entries-read will be inconsistent between master and
      replicas.
      I.e. if no entries were claimed, it would have propagated correctly, but
      if some
      were claimed, then the entries-read field would be inconsistent on the
      replica.
      
      The fix was suggested by guybe7, call streamPropagateGroupID
      unconditionally,
      so that we will normalize entries_read on the replicas. In the past, we
      would
      only set propagate_last_id when NOACK was specified. And in #9127,
      XCLAIM did
      not propagate entries_read in ACK, which would cause entries_read to be
      inconsistent between master and replicas.
      
      Another approach is add another arg to XCLAIM and let it propagate
      entries_read,
      but we decided not to use it. Because we want minimal damage in case
      there's an
      old target and new source (in the worst case scenario, the new source
      doesn't
      recognize XGROUP SETID ... ENTRIES READ and the lag is lost. If we
      change XCLAIM,
      the damage is much more severe).
      
      In this patch, now if the user uses XREADGROUP .. COUNT 1 there will be
      an additional
      overhead of MULTI, EXEC and XGROUPSETID. We assume the extra command in
      case of
      COUNT 1 (4x factor, changing from one XCLAIM to
      MULTI+XCLAIM+XSETID+EXEC), is probably
      ok since reading just one entry is in any case very inefficient (a
      client round trip
      per record), so we're hoping it's not a common case.
      
      Issue was introduced in #9127.
      f17381a3
  5. 22 Feb, 2024 1 commit
  6. 21 Feb, 2024 1 commit
  7. 20 Feb, 2024 1 commit
    • Binbin's avatar
      xinfo-stream add minimum to seen-time, skip logreqres in fuzzer (#13056) · ca5cac99
      Binbin authored
      
      
      Recently I saw in CI that reply-schemas-validator fails here:
      ```
      Failed validating 'minimum' in schema[1]['properties']['groups']['items']['properties']['consumers']['items']['properties']['active-time']:
          {'description': 'Last time this consumer was active (successful '
                          'reading/claiming).',
           'minimum': 0,
           'type': 'integer'}
      
      On instance['groups'][0]['consumers'][0]['active-time']:
          -1729380548878722639
      ```
      
      The reason is that in fuzzer, we may restore corrupted active-time,
      which will cause the reply schema CI to fail.
      
      The fuzzer can cause corrupt the state in many places, which will
      bugs that mess up the reply, so we decided to skip logreqres.
      
      Also, seen-time is the same type as active-time, adding the minimum.
      
      ---------
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ca5cac99
  8. 19 Feb, 2024 1 commit
  9. 04 Feb, 2024 1 commit
  10. 30 Jan, 2024 2 commits
  11. 23 Jan, 2024 1 commit
    • Binbin's avatar
      Allow running WAITAOF in scripts, remove NOSCRIPT flag (#12977) · 85c31e0c
      Binbin authored
      In #11568 we removed the NOSCRIPT flag from commands, e.g. removing
      NOSCRIPT flag from WAIT. Aiming to allow them in scripts and let them
      implicitly behave in the non-blocking way.
      
      This PR remove NOSCRIPT flag from WAITAOF just like WAIT (to be
      symmetrical)).
      And this PR also add BLOCKING flag for WAIT and WAITAOF.
      85c31e0c
  12. 08 Jan, 2024 1 commit
  13. 10 Dec, 2023 1 commit
  14. 12 Oct, 2023 1 commit
    • zhaozhao.zz's avatar
      support XREAD[GROUP] with BLOCK option in scripts (#12596) · 77a65e82
      zhaozhao.zz authored
      In #11568 we removed the NOSCRIPT flag from commands and keep the BLOCKING flag.
      Aiming to allow them in scripts and let them implicitly behave in the non-blocking way.
      
      In that sense, the old behavior was to allow LPOP and reject BLPOP, and the new behavior,
      is to allow BLPOP too, and fail it only in case it ends up blocking.
      So likewise, so far we allowed XREAD and rejected XREAD BLOCK, and we will now allow
      that too, and only reject it if it ends up blocking.
      77a65e82
  15. 10 Oct, 2023 1 commit
  16. 10 Sep, 2023 1 commit
  17. 04 Sep, 2023 1 commit
  18. 31 Aug, 2023 1 commit
  19. 30 Aug, 2023 1 commit
  20. 15 Aug, 2023 1 commit
  21. 05 Aug, 2023 1 commit
  22. 25 Jul, 2023 1 commit
    • nihohit's avatar
      Update request/response policies. (#12417) · 9f512017
      nihohit authored
      changing the response and request policy of a few commands,
      see https://redis.io/docs/reference/command-tips
      
      
      
      1. RANDOMKEY used to have no response policy, which means
        that when sent to multiple shards, the responses should be aggregated.
        this normally applies to commands that return arrays, but since RANDOMKEY
        replies with a simple string, it actually requires a SPECIAL response policy
        (for the client to select just one)
      2. SCAN used to have no response policy, but although the key names part of
        the response can be aggregated, the cursor part certainly can't.
      3. MSETNX had a request policy of MULTI_SHARD and response policy of AGG_MIN,
        but in fact the contract with MSETNX is that when one key exists, it returns 0
        and doesn't set any key, routing it to multiple shards would mean that if one failed
        and another succeeded, it's atomicity is broken and it's impossible to return a valid
        response to the caller.
      Co-authored-by: default avatarShachar Langbeheim <shachlan@amazon.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      9f512017
  23. 20 Jun, 2023 1 commit
  24. 19 Jun, 2023 1 commit
    • Binbin's avatar
      Optimize PSUBSCRIBE and PUNSUBSCRIBE from O(N*M) to O(N) (#12298) · b5106249
      Binbin authored
      In the original implementation, the time complexity of the commands
      is actually O(N*M), where N is the number of patterns the client is
      already subscribed and M is the number of patterns to subscribe to.
      The docs are all wrong about this.
      
      Specifically, because the original client->pubsub_patterns is a list,
      so we need to do listSearchKey which is O(N). In this PR, we change it
      to a dict, so the search becomes O(1).
      
      At the same time, both pubsub_channels and pubsubshard_channels are dicts.
      Changing pubsub_patterns to a dictionary improves the readability and
      maintainability of the code.
      b5106249
  25. 13 Jun, 2023 1 commit
  26. 26 May, 2023 1 commit
    • Binbin's avatar
      Add new loglevel nothing to sentinel-config.json (#12235) · e775b34e
      Binbin authored
      It was missing in #12223, and the reply-schemas daily
      was failing:
      ```
      jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas
      
      Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
          {'oneOf': [{'const': 'debug'},
                     {'const': 'verbose'},
                     {'const': 'notice'},
                     {'const': 'warning'},
                     {'const': 'unknown'}]}
      
      On instance['loglevel']:
          'nothing'
      ```
      e775b34e
  27. 17 May, 2023 1 commit
  28. 10 May, 2023 1 commit
  29. 09 May, 2023 1 commit
  30. 08 May, 2023 1 commit
  31. 27 Apr, 2023 1 commit
    • Binbin's avatar
      Add missing reply schema and coverage tests (#12079) · d659c734
      Binbin authored
      The change in #12018 break the CI (fixed by #12083).
      There are quite a few sentinel commands that are missing both test coverage and also schema.
      
      PR added reply-schema to the following commands:
      - sentinel debug
      - sentinel info-cache
      - sentinel pendding-scripts
      - sentinel reset
      - sentinel simulate-failure
      
      Added some very basic tests for other sentinel commands, just so that they have some coverage.
      - sentinel help
      - sentinel masters
      - sentinel myid
      - sentinel sentinels
      - sentinel slaves
      
      These tests should be improved / replaced in a followup PR.
      d659c734
  32. 20 Apr, 2023 1 commit
  33. 19 Apr, 2023 1 commit
    • Wen Hui's avatar
      Updating reply_schema for sentinal commands (#12018) · b220e6b4
      Wen Hui authored
      Some sentinel subcommands are missing the reply_schema in the json file,
      so add the proper reply_schema part in json file as sentinel replicas commands.
      
      The schema validator was skipping coverage test for sentinel commands, this was initially
      done just in order to focus on redis commands and leave sentinel coverage for later,
      so this check is now removed.
      
      sentinel commands that were missing reply schema:
      * sentinel masters
      * sentinel myid
      * sentinel sentinels <master-name>
      * sentinel slaves (deprecated)  <master-name> 
      b220e6b4
  34. 13 Apr, 2023 1 commit
  35. 29 Mar, 2023 1 commit
    • Itamar Haber's avatar
      Overhauls command summaries and man pages. (#11942) · 0c3b8b7e
      Itamar Haber authored
      This is an attempt to normalize/formalize command summaries.
      
      Main actions performed:
      
      * Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands.
      * Starts with "An internal command...", "A container command...", etc... when applicable.
      * Always uses periods.
      * Refrains from referring to other commands. If this is needed, backquotes should be used for command names.
      * Tries to be very clear about the data type when applicable.
      * Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed."
      * Prefers being terse over verbose.
      * Tries to be consistent.
      0c3b8b7e
  36. 22 Mar, 2023 2 commits
    • Oran Agra's avatar
      ef50118c
    • Igor Malinovskiy's avatar
      Allow clients to report name and version (#11758) · c3b9f2fb
      Igor Malinovskiy authored
      
      
      This PR allows clients to send information about the client library to redis
      to be displayed in CLIENT LIST and CLIENT INFO.
      
      Currently supports:
      `CLIENT [lib-name | lib-ver] <value>`
      Client libraries are expected to pipeline these right after AUTH, and ignore
      the failure in case they're talking to an older version of redis.
      
      These will be shown in CLIENT LIST and CLIENT INFO as:
      * `lib-name` - meant to hold the client library name.
      * `lib-ver` - meant to hold the client library version.
      
      The values cannot contain spaces, newlines and any wild ASCII characters,
      but all other normal chars are accepted, e.g `.`, `=` etc (same as CLIENT NAME).
      
      The RESET command does NOT clear these, but they can be cleared to the
      default by sending a command with a blank string.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      c3b9f2fb
  37. 16 Mar, 2023 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Support for RM_Call on blocking commands (#11568) · d0da0a6a
      Meir Shpilraien (Spielrein) authored
      Allow running blocking commands from within a module using `RM_Call`.
      
      Today, when `RM_Call` is used, the fake client that is used to run command
      is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command
      that it is not allowed to block the client and in case it needs to block, it must
      fallback to some alternative (either return error or perform some default behavior).
      For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block.
      
      All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including
      module commands). When the command invocation finished, Redis asserts that
      the client was not blocked.
      
      This PR introduces the ability to call blocking command using `RM_Call` by
      passing a callback that will be called when the client will get unblocked.
      In order to do that, the user must explicitly say that he allow to perform blocking
      command by passing a new format specifier argument, `K`, to the `RM_Call`
      function. This new flag will tell Redis that it is allow to run blocking command
      and block the client. In case the command got blocked, Redis will return a new
      type of call reply (`REDISMODULE_REPLY_PROMISE`). This call reply indicates
      that the command got blocked and the user can set the on_unblocked handler using
      `RM_CallReplyPromiseSetUnblockHandler`.
      
      When clients gets unblocked, it eventually reaches `processUnblockedClients` function.
      This is where we check if the client is a fake module client and if it is, we call the unblock
      callback instead of performing the usual unblock operations.
      
      **Notice**: `RM_CallReplyPromiseSetUnblockHandler` must be called atomically
      along side the command invocation (without releasing the Redis lock in between).
      In addition, unlike other CallReply types, the promise call reply must be released
      by the module when the Redis GIL is acquired.
      
      The module can abort the execution on the blocking command (if it was not yet
      executed) using `RM_CallReplyPromiseAbort`. the API will return `REDISMODULE_OK`
      on success and `REDISMODULE_ERR` if the operation is already executed.
      **Notice** that in case of misbehave module, Abort might finished successfully but the
      operation will not really be aborted. This can only happened if the module do not respect
      the disconnect callback of the blocked client. 
      For pure Redis commands this can not happened.
      
      ### Atomicity Guarantees
      
      The API promise that the unblock handler will run atomically as an execution unit.
      This means that all the operation performed on the unblock handler will be wrapped
      with a multi exec transaction when replicated to the replica and AOF.
      The API **do not** grantee any other atomicity properties such as when the unblock
      handler will be called. This gives us the flexibility to strengthen the grantees (or not)
      in the future if we will decide that we need a better guarantees.
      
      That said, the implementation **does** provide a better guarantees when performing
      pure Redis blocking command like `BLPOP`. In this case the unblock handler will run
      atomically with the operation that got unblocked (for example, in case of `BLPOP`, the
      unblock handler will run atomically with the `LPOP` operation that run when the command
      got unblocked). This is an implementation detail that might be change in the future and the
      module writer should not count on that.
      
      ### Calling blocking commands while running on script mode (`S`)
      
      `RM_Call` script mode (`S`) was introduced on #0372. It is used for usecases where the
      command that was invoked on `RM_Call` comes from a user input and we want to make
      sure the user will not run dangerous commands like `shutdown`. Some command, such
      as `BLPOP`, are marked with `NO_SCRIPT` flag, which means they will not be allowed on
      script mode. Those commands are marked with  `NO_SCRIPT` just because they are
      blocking commands and not because they are dangerous. Now that we can run blocking
      commands on RM_Call, there is no real reason not to allow such commands on script mode.
      
      The underline problem is that the `NO_SCRIPT` flag is abused to also mark some of the
      blocking commands (notice that those commands know not to block the client if it is not
      allowed to do so, and have a fallback logic to such cases. So even if those commands
      were not marked with `NO_SCRIPT` flag, it would not harm Redis, and today we can
      already run those commands within multi exec).
      
      In addition, not all blocking commands are marked with `NO_SCRIPT` flag, for example
      `blmpop` are not marked and can run from within a script.
      
      Those facts shows that there are some ambiguity about the meaning of the `NO_SCRIPT`
      flag, and its not fully clear where it should be use.
      
      The PR suggest that blocking commands should not be marked with `NO_SCRIPT` flag,
      those commands should handle `CLIENT_DENY_BLOCKING` flag and only block when
      it's safe (like they already does today). To achieve that, the PR removes the `NO_SCRIPT`
      flag from the following commands:
      * `blmove`
      * `blpop`
      * `brpop`
      * `brpoplpush`
      * `bzpopmax`
      * `bzpopmin`
      * `wait`
      
      This might be considered a breaking change as now, on scripts, instead of getting
      `command is not allowed from script` error, the user will get some fallback behavior
      base on the command implementation. That said, the change matches the behavior
      of scripts and multi exec with respect to those commands and allow running them on
      `RM_Call` even when script mode is used.
      
      ### Additional RedisModule API and changes
      
      * `RM_BlockClientSetPrivateData` - Set private data on the blocked client without the
        need to unblock the client. This allows up to set the promise CallReply as the private
        data of the blocked client and abort it if the client gets disconnected.
      * `RM_BlockClientGetPrivateData` - Return the current private data set on a blocked client.
        We need it so we will have access to this private data on the disconnect callback.
      * On RM_Call, the returned reply will be added to the auto memory context only if auto
        memory is enabled, this allows us to keep the call reply for longer time then the context
        lifetime and does not force an unneeded borrow relationship between the CallReply and
        the RedisModuleContext.
      d0da0a6a
  38. 14 Mar, 2023 1 commit
    • Slava Koyfman's avatar
      Implementing the WAITAOF command (issue #10505) (#11713) · 9344f654
      Slava Koyfman authored
      
      
      Implementing the WAITAOF functionality which would allow the user to
      block until a specified number of Redises have fsynced all previous write
      commands to the AOF.
      
      Syntax: `WAITAOF <num_local> <num_replicas> <timeout>`
      Response: Array containing two elements: num_local, num_replicas
      num_local is always either 0 or 1 representing the local AOF on the master.
      num_replicas is the number of replicas that acknowledged the a replication
      offset of the last write being fsynced to the AOF.
      
      Returns an error when called on replicas, or when called with non-zero
      num_local on a master with AOF disabled, in all other cases the response
      just contains number of fsync copies.
      
      Main changes:
      * Added code to keep track of replication offsets that are confirmed to have
        been fsynced to disk.
      * Keep advancing master_repl_offset even when replication is disabled (and
        there's no replication backlog, only if there's an AOF enabled).
        This way we can use this command and it's mechanisms even when replication
        is disabled.
      * Extend REPLCONF ACK to `REPLCONF ACK <ofs> FACK <ofs>`, the FACK
        will be appended only if there's an AOF on the replica, and already ignored on
        old masters (thus backwards compatible)
      * WAIT now no longer wait for the replication offset after your last command, but
        rather the replication offset after your last write (or read command that caused
        propagation, e.g. lazy expiry).
      
      Unrelated changes:
      * WAIT command respects CLIENT_DENY_BLOCKING (not just CLIENT_MULTI)
      
      Implementation details:
      * Add an atomic var named `fsynced_reploff_pending` that's updated
        (usually by the bio thread) and later copied to the main `fsynced_reploff`
        variable (only if the AOF base file exists).
        I.e. during the initial AOF rewrite it will not be used as the fsynced offset
        since the AOF base is still missing.
      * Replace close+fsync bio job with new BIO_CLOSE_AOF (AOF specific)
        job that will also update fsync offset the field.
      * Handle all AOF jobs (BIO_CLOSE_AOF, BIO_AOF_FSYNC) in the same bio
        worker thread, to impose ordering on their execution. This solves a
        race condition where a job could set `fsynced_reploff_pending` to a higher
        value than another pending fsync job, resulting in indicating an offset
        for which parts of the data have not yet actually been fsynced.
        Imposing an ordering on the jobs guarantees that fsync jobs are executed
        in increasing order of replication offset.
      * Drain bio jobs when switching `appendfsync` to "always"
        This should prevent a write race between updates to `fsynced_reploff_pending`
        in the main thread (`flushAppendOnlyFile` when set to ALWAYS fsync), and
        those done in the bio thread.
      * Drain the pending fsync when starting over a new AOF to avoid race conditions
        with the previous AOF offsets overriding the new one (e.g. after switching to
        replicate from a new master).
      * Make sure to update the fsynced offset at the end of the initial AOF rewrite.
        a must in case there are no additional writes that trigger a periodic fsync,
        specifically for a replica that does a full sync.
      
      Limitations:
      It is possible to write a module and a Lua script that propagate to the AOF and doesn't
      propagate to the replication stream. see REDISMODULE_ARGV_NO_REPLICAS and luaRedisSetReplCommand.
      These features are incompatible with the WAITAOF command, and can result
      in two bad cases. The scenario is that the user executes command that only
      propagates to AOF, and then immediately
      issues a WAITAOF, and there's no further writes on the replication stream after that.
      1. if the the last thing that happened on the replication stream is a PING
        (which increased the replication offset but won't trigger an fsync on the replica),
        then the client would hang forever (will wait for an fack that the replica will never
        send sine it doesn't trigger any fsyncs).
      2. if the last thing that happened is a write command that got propagated properly,
        then WAITAOF will be released immediately, without waiting for an fsync (since
        the offset didn't change)
      
      Refactoring:
      * Plumbing to allow bio worker to handle multiple job types
        This introduces infrastructure necessary to allow BIO workers to
        not have a 1-1 mapping of worker to job-type. This allows in the
        future to assign multiple job types to a single worker, either as
        a performance/resource optimization, or as a way of enforcing
        ordering between specific classes of jobs.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      9344f654