1. 02 Dec, 2020 1 commit
  2. 01 Dec, 2020 3 commits
    • Madelyn Olson's avatar
      Getset fix (#8118) · 69b7113b
      Madelyn Olson authored
      
      
      * Fixed SET GET executing on wrong type
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      69b7113b
    • sundb's avatar
      Improve dbid range check for SELECT, MOVE, COPY (#8085) · 3ba2281f
      sundb authored
      SELECT used to read the index into a `long` variable, and then pass it to a function
      that takes an `int`, possibly causing an overflow before the range check.
      
      Now all these commands use better and cleaner range check, and that also results in
      a slight change of the error response in case of an invalid database index.
      
      SELECT:
      in the past it would have returned either `-ERR invalid DB index` (if not a number),
      or `-ERR DB index is out of range` (if not between 1..16 or alike).
      now it'll return either `-ERR value is out of range` (if not a number), or
      `-ERR value is out of range, value must between -2147483648 and 2147483647`
      (if not in the range for an int), or `-ERR DB index is out of range`
      (if not between 0..16 or alike)
      
      
      MOVE:
      in the past it would only fail with `-ERR index out of range` no matter the reason.
      now return the same errors as the new ones for SELECT mentioned above.
      (i.e. unlike for SELECT even for a value like 17 we changed the error message)
      
      COPY:
      doesn't really matter how it behaved in the past (new command), new behavior is
      like the above two.
      3ba2281f
    • Itamar Haber's avatar
      Adds pub/sub channel patterns to ACL (#7993) · c1b1e8c3
      Itamar Haber authored
      Fixes #7923.
      
      This PR appropriates the special `&` symbol (because `@` and `*` are taken),
      followed by a literal value or pattern for describing the Pub/Sub patterns that
      an ACL user can interact with. It is similar to the existing key patterns
      mechanism in function (additive) and implementation (copy-pasta). It also adds
      the allchannels and resetchannels ACL keywords, naturally.
      
      The default user is given allchannels permissions, whereas new users get
      whatever is defined by the acl-pubsub-default configuration directive. For
      backward compatibility in 6.2, the default of this directive is allchannels but
      this is likely to be changed to resetchannels in the next major version for
      stronger default security settings.
      
      Unless allchannels is set for the user, channel access permissions are checked
      as follows :
      * Calls to both PUBLISH and SUBSCRIBE will fail unless a pattern matching the
        argumentative channel name(s) exists for the user.
      * Calls to PSUBSCRIBE will fail unless the pattern(s) provided as an argument
        literally exist(s) in the user's list.
      
      Such failures are logged to the ACL log.
      
      Runtime changes to channel permissions for a user with existing subscribing
      clients cause said clients to disconnect unless the new permissions permit the
      connections to continue. Note, however, that PSUBSCRIBErs' patterns are matched
      literally, so given the change bar:* -> b*, pattern subscribers to bar:* will be
      disconnected.
      
      Notes/questions:
      * UNSUBSCRIBE, PUNSUBSCRIBE and PUBSUB remain unprotected due to lack of reasons
        for touching them.
      c1b1e8c3
  3. 29 Nov, 2020 1 commit
    • guybe7's avatar
      XPENDING with IDLE (#7972) · ada2ac9a
      guybe7 authored
      Used to filter stream pending entries by their idle-time,
      useful for XCLAIMing entries that have not been processed
      for some time
      ada2ac9a
  4. 23 Nov, 2020 1 commit
  5. 18 Nov, 2020 1 commit
    • guybe7's avatar
      EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica (#8016) · f8ae9917
      guybe7 authored
      The bug was introduced by #5021 which only attempted avoid EXIST on an
      already expired key from returning 1 on a replica.
      
      Before that commit, dbExists was used instead of
      lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)
      
      Other than that, this commit fixes OBJECT to also come empty handed on
      expired keys in replica.
      
      And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
      the key regardless of it's expired state)
      f8ae9917
  6. 17 Nov, 2020 3 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025) · d87a0d02
      Meir Shpilraien (Spielrein) authored
      
      
      Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
      the caller, who executes the command in this context, expects a reply.
      
      Today, LUA and MULTI have a special (and different) treatment to blocking commands:
      
      LUA   - Most commands are marked with no-script flag which are checked when executing
      and command from LUA, commands that are not marked (like XREAD) verify that their
      blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
      MULTI - Command that is going to block, first verify that the client is not inside
      multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
      return a result which is a match to the empty key with no timeout (for example blpop
      inside MULTI will act as lpop)
      For modules that perform RM_Call with blocking command, the returned results type is
      REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.
      
      Disadvantages of the current state are:
      
      No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
      Module can not safely execute blocking command (and get reply or error).
      Though It is true that modules are not like LUA or MULTI and should be smarter not
      to execute blocking commands on RM_Call, sometimes you want to execute a command base
      on client input (for example if you create a module that provides a new scripting
      language like javascript or python).
      While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
      REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
      check if the command came from another module using RM_Call. So there is no way
      for a module to know not to block another module RM_Call execution.
      
      This commit adds a way to unify the treatment for blocking clients by introducing
      a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
      turned on to signify that the client should not be blocked. A blocking command
      verifies that the flag is turned off before blocking. If a blocking command sees
      that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
      which are matches to empty key with no timeout (as MULTI does today).
      
      The new flag is checked on the following commands:
      
      List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
      Zset blocking commands: BZPOPMIN, BZPOPMAX
      Stream blocking commands: XREAD, XREADGROUP
      SUBSCRIBE, PSUBSCRIBE, MONITOR
      In addition, the new flag is turned on inside the AOF client, we do not want to
      block the AOF client to prevent deadlocks and commands ordering issues (and there
      is also an existing assert in the code that verifies it).
      
      To keep backward compatibility on LUA, all the no-script flags on existing commands
      were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.
      
      To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
      We added a special treatment on those commands to allow executing them on MULTI.
      
      The only backward compatibility issue that this PR introduces is that now MONITOR
      is not allowed inside MULTI.
      
      Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
      or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      d87a0d02
    • thomaston's avatar
      ZREVRANGEBYSCORE Optimization for out of range offset (#5773) · 39f716a1
      thomaston authored
      
      
      ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
      When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      39f716a1
    • swamp0407's avatar
      Add COPY command (#7953) · ea7cf737
      swamp0407 authored
      
      
      Syntax:
      COPY <key> <new-key> [DB <dest-db>] [REPLACE]
      
      No support for module keys yet.
      
      Co-authored-by: tmgauss
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ea7cf737
  7. 16 Nov, 2020 1 commit
    • chenyangyang's avatar
      Modules callbacks for lazy free effort, and unlink (#7912) · c1aaad06
      chenyangyang authored
      Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
      and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
      Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
      asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
      may soon be freed by a background thread.
      
      Add `lazyfreed_objects` info field, which represents the number of objects that have been
      lazyfreed since redis was started.
      
      Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
      `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
      which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
      c1aaad06
  8. 15 Nov, 2020 1 commit
  9. 11 Nov, 2020 1 commit
  10. 10 Nov, 2020 1 commit
  11. 09 Nov, 2020 1 commit
    • nitaicaro's avatar
      Extend client tracking tests (#7998) · 19c29b60
      nitaicaro authored
      Test support for the new map, null and push message types. Map objects are parsed as a list of lists of key value pairs.
      for instance: user => john password => 123
      
      will be parsed to the following TCL list:
      
      {{user john} {password 123}}
      
      Also added the following tests:
      
      Redirection still works with RESP3
      
      Able to use a RESP3 client as a redirection client
      
      No duplicate invalidation messages when turning BCAST mode on after normal tracking
      
      Server is able to evacuate enough keys when num of keys surpasses limit by more than defined initial effort
      
      Different clients using different protocols can track the same key
      
      OPTOUT tests
      
      OPTIN tests
      
      Clients can redirect to the same connection
      
      tracking-redir-broken test
      
      HELLO 3 checks
      
      Invalidation messages still work when using RESP3, with and without redirection
      
      Switching to RESP3 doesn't disturb previous tracked keys
      
      Tracking info is correct
      
      Flushall and flushdb produce invalidation messages
      
      These tests achieve 100% line coverage for tracking.c using lcov.
      19c29b60
  12. 08 Nov, 2020 1 commit
  13. 05 Nov, 2020 1 commit
    • Yossi Gottlieb's avatar
      Add RESET command. (#7982) · 1fd456f9
      Yossi Gottlieb authored
      
      
      Perform full reset of all client connection states, is if the client was
      disconnected and re-connected. This affects:
      
      * MULTI state
      * Watched keys
      * MONITOR mode
      * Pub/Sub subscription
      * ACL/Authenticated state
      * Client tracking state
      * Cluster read-only/asking state
      * RESP version (reset to 2)
      * Selected database
      * CLIENT REPLY state
      
      The response is +RESET to make it easily distinguishable from other
      responses.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      1fd456f9
  14. 04 Nov, 2020 1 commit
    • Yossi Gottlieb's avatar
      Fix test failure on slower systems. · 2faa0f19
      Yossi Gottlieb authored
      Not disabling save, slower systems begun background save that did not
      complete in time, resulting with SAVE failing with "ERR Background save
      already in progress".
      2faa0f19
  15. 03 Nov, 2020 1 commit
  16. 28 Oct, 2020 1 commit
  17. 22 Oct, 2020 2 commits
  18. 18 Oct, 2020 1 commit
    • Wen Hui's avatar
      support NOMKSTREAM option in xadd command (#7910) · f328194d
      Wen Hui authored
      introduces a NOMKSTREAM option for xadd command, this would be useful for some
      use cases when we do not want to create new stream by default:
      
      XADD key [MAXLEN [~|=] <count>] [NOMKSTREAM] <ID or *> [field value] [field value]
      f328194d
  19. 11 Oct, 2020 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Add Module API for version and compatibility checks (#7865) · adc3183c
      Meir Shpilraien (Spielrein) authored
      
      
      * Introduce a new API's: RM_GetContextFlagsAll, and
      RM_GetKeyspaceNotificationFlagsAll that will return the
      full flags mask of each feature. The module writer can
      check base on this value if the Flags he needs are
      supported or not.
      
      * For each flag, introduce a new value on redismodule.h,
      this value represents the LAST value and should be there
      as a reminder to update it when a new value is added,
      also it will be used in the code to calculate the full
      flags mask (assuming flags are incrementally increasing).
      In addition, stated that the module writer should not use
      the LAST flag directly and he should use the GetFlagAll API's.
      
      * Introduce a new API: RM_IsSubEventSupported, that returns for a given
      event and subevent, whether or not the subevent supported.
      
      * Introduce a new macro RMAPI_FUNC_SUPPORTED(func) that returns whether
      or not a function API is supported by comparing it to NULL.
      
      * Introduce a new API: int RM_GetServerVersion();, that will return the
      current Redis version in the format 0x00MMmmpp; e.g. 0x00060008;
      
      * Changed unstable version from 999.999.999 to 255.255.255
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      adc3183c
    • Yossi Gottlieb's avatar
      Module API: Add RM_GetClientCertificate(). (#7866) · 0aec98dc
      Yossi Gottlieb authored
      This API function makes it possible to retrieve the X.509 certificate
      used by clients to authenticate TLS connections.
      0aec98dc
    • Yossi Gottlieb's avatar
      Modules: Add RM_GetDetachedThreadSafeContext(). (#7886) · 907da058
      Yossi Gottlieb authored
      The main motivation here is to provide a way for modules to create a
      single, global context that can be used for logging.
      
      Currently, it is possible to obtain a thread-safe context that is not
      attached to any blocked client by using `RM_GetThreadSafeContext`.
      However, the attached context is not linked to the module identity so
      log messages produced are not tagged with the module name.
      
      Ideally we'd fix this in `RM_GetThreadSafeContext` itself but as it
      doesn't accept the current context as an argument there's no way to do
      that in a backwards compatible manner.
      907da058
    • Yossi Gottlieb's avatar
      Modules: add RM_GetCommandKeys(). · 7d117d75
      Yossi Gottlieb authored
      This is essentially the same as calling COMMAND GETKEYS but provides a
      more efficient interface that can be used in every context (i.e. not a
      Redis command).
      7d117d75
  20. 08 Oct, 2020 2 commits
    • Felipe Machado's avatar
      Adds new pop-push commands (LMOVE, BLMOVE) (#6929) · c3f9e017
      Felipe Machado authored
      
      
      Adding [B]LMOVE <src> <dst> RIGHT|LEFT RIGHT|LEFT. deprecating [B]RPOPLPUSH.
      
      Note that when receiving a BRPOPLPUSH we'll still propagate an RPOPLPUSH,
      but on BLMOVE RIGHT LEFT we'll propagate an LMOVE
      
      improvement to existing tests
      - Replace "after 1000" with "wait_for_condition" when wait for
        clients to block/unblock.
      - Add a pre-existing element to target list on basic tests so
        that we can check if the new element was added to the correct
        side of the list.
      - check command stats on the replica to make sure the right
        command was replicated
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      c3f9e017
    • Madelyn Olson's avatar
  21. 05 Oct, 2020 1 commit
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · bea40e6a
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      bea40e6a
  22. 02 Oct, 2020 1 commit
  23. 01 Oct, 2020 1 commit
  24. 30 Sep, 2020 1 commit
    • nitaicaro's avatar
      Fixed Tracking test “The other connection is able to get invalidations” (#7871) · 8fb89a57
      nitaicaro authored
      
      
      PROBLEM:
      
      [$rd1 read] reads invalidation messages one by one, so it's never going to see the second invalidation message produced after INCR b, whether or not it exists. Adding another read will block incase no invalidation message is produced.
      
      FIX:
      
      We switch the order of "INCR a" and "INCR b" - now "INCR b" comes first. We still only read the first invalidation message produces. If an invalidation message is wrongly produces for b - then it will be produced before that of a, since "INCR b" comes before "INCR a".
      Co-authored-by: default avatarNitai Caro <caronita@amazon.com>
      8fb89a57
  25. 27 Sep, 2020 1 commit
  26. 24 Sep, 2020 3 commits
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 57709c4b
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      57709c4b
    • valentinogeron's avatar
      Stream: Inconsistency between master and replica some XREADGROUP case (#7526) · 795c454d
      valentinogeron authored
      XREADGROUP auto-creates the consumer inside the consumer group the
      first time it saw it.
      When XREADGROUP is being used with NOACK option, the message will not
      be added into the client's PEL and XGROUP SETID would be propagated.
      When the replica gets the XGROUP SETID it will only update the last delivered
      id of the group, but will not create the consumer.
      
      So, in this commit XGROUP CREATECONSUMER is being added.
      Command pattern: XGROUP CREATECONSUMER <key> <group> <consumer>.
      
      When NOACK option is being used, createconsumer command would be
      propagated as well.
      
      In case of AOFREWRITE, consumer with an empty PEL would be saved with
      XGROUP CREATECONSUMER whereas consumer with pending entries would be
      saved with XCLAIM
      795c454d
    • bodong.ybd's avatar
      Add ZINTER/ZUNION command · e08bf166
      bodong.ybd authored
      Syntax: ZINTER/ZUNION numkeys key [key ...] [WEIGHTS weight [weight ...]]
      [AGGREGATE SUM|MIN|MAX] [WITHSCORES]
      
      see #7624
      e08bf166
  27. 23 Sep, 2020 1 commit
  28. 20 Sep, 2020 1 commit
  29. 17 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Remove tmp rdb file in background thread (#7762) · b002d2b4
      Wang Yuan authored
      We're already using bg_unlink in several places to delete the rdb file in the background,
      and avoid paying the cost of the deletion from our main thread.
      This commit uses bg_unlink to remove the temporary rdb file in the background too.
      
      However, in case we delete that rdb file just before exiting, we don't actually wait for the
      background thread or the main thread to delete it, and just let the OS clean up after us.
      i.e. we open the file, unlink it and exit with the fd still open.
      
      Furthermore, rdbRemoveTempFile can be called from a thread and was using snprintf which is
      not async-signal-safe, we now use ll2string instead.
      b002d2b4