1. 17 Nov, 2020 4 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Unified MULTI, LUA, and RM_Call with respect to blocking commands (#8025) · d87a0d02
      Meir Shpilraien (Spielrein) authored
      
      
      Blocking command should not be used with MULTI, LUA, and RM_Call. This is because,
      the caller, who executes the command in this context, expects a reply.
      
      Today, LUA and MULTI have a special (and different) treatment to blocking commands:
      
      LUA   - Most commands are marked with no-script flag which are checked when executing
      and command from LUA, commands that are not marked (like XREAD) verify that their
      blocking mode is not used inside LUA (by checking the CLIENT_LUA client flag).
      MULTI - Command that is going to block, first verify that the client is not inside
      multi (by checking the CLIENT_MULTI client flag). If the client is inside multi, they
      return a result which is a match to the empty key with no timeout (for example blpop
      inside MULTI will act as lpop)
      For modules that perform RM_Call with blocking command, the returned results type is
      REDISMODULE_REPLY_UNKNOWN and the caller can not really know what happened.
      
      Disadvantages of the current state are:
      
      No unified approach, LUA, MULTI, and RM_Call, each has a different treatment
      Module can not safely execute blocking command (and get reply or error).
      Though It is true that modules are not like LUA or MULTI and should be smarter not
      to execute blocking commands on RM_Call, sometimes you want to execute a command base
      on client input (for example if you create a module that provides a new scripting
      language like javascript or python).
      While modules (on modules command) can check for REDISMODULE_CTX_FLAGS_LUA or
      REDISMODULE_CTX_FLAGS_MULTI to know not to block the client, there is no way to
      check if the command came from another module using RM_Call. So there is no way
      for a module to know not to block another module RM_Call execution.
      
      This commit adds a way to unify the treatment for blocking clients by introducing
      a new CLIENT_DENY_BLOCKING client flag. On LUA, MULTI, and RM_Call the new flag
      turned on to signify that the client should not be blocked. A blocking command
      verifies that the flag is turned off before blocking. If a blocking command sees
      that the CLIENT_DENY_BLOCKING flag is on, it's not blocking and return results
      which are matches to empty key with no timeout (as MULTI does today).
      
      The new flag is checked on the following commands:
      
      List blocking commands: BLPOP, BRPOP, BRPOPLPUSH, BLMOVE,
      Zset blocking commands: BZPOPMIN, BZPOPMAX
      Stream blocking commands: XREAD, XREADGROUP
      SUBSCRIBE, PSUBSCRIBE, MONITOR
      In addition, the new flag is turned on inside the AOF client, we do not want to
      block the AOF client to prevent deadlocks and commands ordering issues (and there
      is also an existing assert in the code that verifies it).
      
      To keep backward compatibility on LUA, all the no-script flags on existing commands
      were kept untouched. In addition, a LUA special treatment on XREAD and XREADGROUP was kept.
      
      To keep backward compatibility on MULTI (which today allows SUBSCRIBE, and PSUBSCRIBE).
      We added a special treatment on those commands to allow executing them on MULTI.
      
      The only backward compatibility issue that this PR introduces is that now MONITOR
      is not allowed inside MULTI.
      
      Tests were added to verify blocking commands are not blocking the client on LUA, MULTI,
      or RM_Call. Tests were added to verify the module can check for CLIENT_DENY_BLOCKING flag.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      d87a0d02
    • thomaston's avatar
      ZREVRANGEBYSCORE Optimization for out of range offset (#5773) · 39f716a1
      thomaston authored
      
      
      ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]
      When the offset is too large, the query is very slow. Especially when the offset is greater than the length of zset it is easy to determine whether the offset is greater than the length of zset at first, and If it exceed the length of zset, then return directly.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      39f716a1
    • Yossi Gottlieb's avatar
      Improve and clean up supervised process support. (#8036) · d638b058
      Yossi Gottlieb authored
      * Configuration file default should now be "auto".
      * Expose "process_supervised" as an info field.
      * Log messages improvements (clarify required systemd config, report
        auto-detected supervision mode, etc.)
      * Set server.supervised properly, so it can take precedence of
        "daemonize" configuration.
      * Produce clear warning if systemd is detected/requested but executable
        is compiled without support for it, instead of silently ignoring.
      * Handle systemd notification error on startup, and turn off supervised
        mode if it failed.
      d638b058
    • swamp0407's avatar
      Add COPY command (#7953) · ea7cf737
      swamp0407 authored
      
      
      Syntax:
      COPY <key> <new-key> [DB <dest-db>] [REPLACE]
      
      No support for module keys yet.
      
      Co-authored-by: tmgauss
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ea7cf737
  2. 16 Nov, 2020 2 commits
    • Oran Agra's avatar
      Fix memory leaks in newly added ZDIFF (#8056) · 9812e889
      Oran Agra authored
      9812e889
    • chenyangyang's avatar
      Modules callbacks for lazy free effort, and unlink (#7912) · c1aaad06
      chenyangyang authored
      Add two optional callbacks to the RedisModuleTypeMethods structure, which is `free_effort`
      and `unlink`. the `free_effort` callback indicates the effort required to free a module memory.
      Currently, if the effort exceeds LAZYFREE_THRESHOLD, the module memory may be released
      asynchronously. the `unlink` callback indicates the key has been removed from the DB by redis, and
      may soon be freed by a background thread.
      
      Add `lazyfreed_objects` info field, which represents the number of objects that have been
      lazyfreed since redis was started.
      
      Add `RM_GetTypeMethodVersion` API, which return the current redis-server runtime value of
      `REDISMODULE_TYPE_METHOD_VERSION`. You can use that when calling `RM_CreateDataType` to know
      which fields of RedisModuleTypeMethods are gonna be supported and which will be ignored.
      c1aaad06
  3. 15 Nov, 2020 1 commit
  4. 13 Nov, 2020 1 commit
  5. 12 Nov, 2020 2 commits
    • Itamar Haber's avatar
      Adds user parsing to redis-cli URIs (#8048) · 92ec5925
      Itamar Haber authored
      92ec5925
    • Yash Ladha's avatar
      cleanup: move list pop logic to single function (#7997) · c170365d
      Yash Ladha authored
      BLPOP when there are elements in the list works in the same way as LPOP
      does. Due to this they also does the same repetitive action and logic
      for the same is written at two different places. This is a bad code
      practice as the one needs the context to change the BLPOP list pop code
      as well when the LPOP code gets changed.
      
      Separated the generic logic from LPOP to a function that is being used
      by the BLPOP code as well.
      c170365d
  6. 11 Nov, 2020 3 commits
  7. 10 Nov, 2020 3 commits
  8. 09 Nov, 2020 1 commit
  9. 08 Nov, 2020 3 commits
  10. 05 Nov, 2020 4 commits
    • Wen Hui's avatar
      Debug Populate: Avoid server crash when passing negative value for key and value size (#8018) · 25436778
      Wen Hui authored
      * Debug Populate: Add checks for count and keysize to avoid crash
      
      * provide getRangeLongFromObjectOrReply and getPositiveLongFromObjectOrReply for range check
      25436778
    • Yossi Gottlieb's avatar
      Fix crash log output on ARM. (#8020) · 7e4325cb
      Yossi Gottlieb authored
      7e4325cb
    • Oran Agra's avatar
      Better INFO fields to track diskless and disk-based replication progress (#7981) · 7ace7231
      Oran Agra authored
      Expose new `loading_rdb_used_mem` showing the used memory of the server
      that saved the RDB file we're currently using.
      This is useful in diskless replication when the total size of the rdb is
      unkown, and can be used as a rought estimation of progres.
      
      Use that new field to calculate the "user friendly"
      `loading_loaded_perc` and `loading_eta_seconds`.
      
      Expose `master_sync_total_bytes` and `master_sync_total_bytes` to complement
      on the existing `master_sync_total_bytes` (which cannot be used on its own
      to calculate progress).
      
      Add "user friendly" field for `master_sync_perc`
      7ace7231
    • Yossi Gottlieb's avatar
      Add RESET command. (#7982) · 1fd456f9
      Yossi Gottlieb authored
      
      
      Perform full reset of all client connection states, is if the client was
      disconnected and re-connected. This affects:
      
      * MULTI state
      * Watched keys
      * MONITOR mode
      * Pub/Sub subscription
      * ACL/Authenticated state
      * Client tracking state
      * Cluster read-only/asking state
      * RESP version (reset to 2)
      * Selected database
      * CLIENT REPLY state
      
      The response is +RESET to make it easily distinguishable from other
      responses.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      1fd456f9
  11. 04 Nov, 2020 5 commits
    • Tomasz Poradowski's avatar
      rdb: corrected RedisModuleIO initialization point (#8014) · d8fbd3a8
      Tomasz Poradowski authored
      
      
      - rdbSaveSingleModuleAux() used RedisModuleIO's "bytes" field for
        tracking written bytes before calling moduleInitIOContext() which sets
        "bytes" to zero
      - rdbSaveObject() re-initialized RedisModuleIO too late
      
      This return value is not used at the moment since it's only tested
      against -1, and the actual byte count isn't used yet.
      Co-authored-by: default avatarTomasz Poradowski <tomasz.poradowski@generiscorp.com>
      d8fbd3a8
    • filipe oliveira's avatar
      Enable specifying TLS ciphers(suites) in redis-cli/redis-benchmark (#8005) · 10b50069
      filipe oliveira authored
      
      
      Enable specifying the preferred ciphers and/or ciphersuites for redis-cli/redis-benchmark.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      10b50069
    • Egor Seredin's avatar
      Allow '\0' inside of result of sdscatvprintf, and efficiency improvements (#6260) · f4ca3d87
      Egor Seredin authored
      This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2");
      
      On large string, the previous code would incrementally retry to double the output buffer.
      now it uses the the return value of snprintf and grows to the right size in one step.
      
      and also avoids an excessive strlen in sdscat at the end.
      f4ca3d87
    • Wen Hui's avatar
      redis-cli cluster import support source and target that need auth (#7994) · 639b73cd
      Wen Hui authored
      Make it possible for redis-cli cluster import to work with source and
      target that require AUTH.
      
      Adding two different flags --cluster-from-user, --cluster-from-pass
      and --cluster-askpass for source node authentication.
      Also for target authentication, using existing --user and --pass flag.
      
      Example:
      
      ./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --pass 1234 --user default --cluster-from-user default --cluster-from-pass 123456
      
      ./redis-cli --cluster import 127.0.0.1:7000 --cluster-from 127.0.0.1:6379 --askpass --cluster-from-user default --cluster-from-askpass
      639b73cd
    • Oran Agra's avatar
      Add maxclients and cluster_connections to INFO CLIENTS (#7979) · a698a639
      Oran Agra authored
      Few config settings are also reflected by the INFO command.
      these are mainly ones that are important for either an instant view of
      the server status (to compare a metric to it's limit config),
      Important configurations that are necessary in the crash log (which
      currently doesn't print the config),
      And things that are important for monitoring solutions (such as
      Prometheus), which rely on INFO to collect their data.
      
      Add cluster_connections to INFO CLUSTER:
      This makes it possible to be combined together with connected_clients
      and connected_slaves and be matched against maxclients
      a698a639
  12. 03 Nov, 2020 3 commits
  13. 02 Nov, 2020 1 commit
    • guybe7's avatar
      Modules: Improve timer accuracy (#7987) · 1a91a270
      guybe7 authored
      The bug occurs when 'callback' re-registers itself to a point
      in the future and the execution time in non-negligible:
      'now' refers to time BEFORE callback was executed and is used
      to calculate 'next_period'.
      We must get the actual current time when calculating 'next_period'
      1a91a270
  14. 28 Oct, 2020 7 commits
    • yoav-steinberg's avatar
      Add local address to CLIENT LIST, and a CLIENT KILL filter. (#7913) · 84b3c18f
      yoav-steinberg authored
      Useful when you want to know through which bind address the client connected to
      the server in case of multiple bind addresses.
      
      - Adding `laddr` field to CLIENT list showing the local (bind) address.
      - Adding `LADDR` option to CLIENT KILL to kill all the clients connected
        to a specific local address.
      - Refactoring to share code.
      84b3c18f
    • Oran Agra's avatar
      Optionally (default) fail to start if requested bind address is not available (#7936) · 441bfa2d
      Oran Agra authored
      Background:
      #3467 (redis 4.0.0), started ignoring ENOPROTOOPT, but did that only for
      the default bind (in case bind config wasn't explicitly set).
      #5598 (redis 5.0.3), added that for bind addresses explicitly set
      (following bug reports in Debian for redis 4.0.9 and 5.0.1), it
      also ignored a bunch of other errors like EPROTONOSUPPORT which was
      requested in #3894, and also added EADDRNOTAVAIL (wasn't clear why).
      
      This (ignoring EADDRNOTAVAIL) makes redis start successfully, even if a
      certain network interface isn't up yet , in which case we rather redis
      fail and will be re-tried when the NIC is up, see #7933.
      
      However, it turns out that when IPv6 is disabled (supported but unused),
      the error we're getting is EADDRNOTAVAIL. and in many systems the
      default config file tries to bind to localhost for both v4 and v6 and
      would like to silently ignore the error on v6 if disabled.
      This means that we sometimes want to ignore EADDRNOTAVAIL and other times
      we wanna fail.
      
      So this commit changes these main things:
      1. Ignore all the errors we ignore for both explicitly requested bind
         address and a default implicit one.
      2. Add a '-' prefix to allow EADDRNOTAVAIL be ignored (by default that's
         different than the previous behavior).
      3. Restructure that function in a more readable and maintainable way see
         below.
      4. Make the default behavior of listening to all achievable by setting
        a bind config directive to * (previously only possible by omitting
        it)
      5. document everything.
      
      The old structure of this function was that even if there are no bind
      addresses requested, the loop that runs though the bind addresses runs
      at least once anyway!
      In that one iteration of the loop it binds to both v4 and v6 addresses,
      handles errors for each of them separately, and then eventually at the
      if-else chain, handles the error of the last bind attempt again!
      This was very hard to read and very error prone to maintain, instead now
      when the bind info is missing we create one with two entries, and run
      the simple loop twice.
      441bfa2d
    • Madelyn Olson's avatar
      d310beb4
    • Madelyn Olson's avatar
      411bcf1a
    • Wen Hui's avatar
    • sundb's avatar
      69871760
    • filipe oliveira's avatar
      TLS Support for redis-benchmark (#7959) · 39436b21
      filipe oliveira authored
      39436b21