1. 06 Jan, 2021 2 commits
    • Wen Hui's avatar
      fix memory leak in processInlineBuffer error handling code (#8295) · cfcd0fa6
      Wen Hui authored
      This code path is normally executed only when v6.0 and above replicates from v2.4
      cfcd0fa6
    • guybe7's avatar
      Add XAUTOCLAIM (#7973) · 714e103a
      guybe7 authored
      
      
      New command: XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT <count>] [JUSTID]
      
      The purpose is to claim entries from a stale consumer without the usual
      XPENDING+XCLAIM combo which takes two round trips.
      
      The syntax for XAUTOCLAIM is similar to scan: A cursor is returned (streamID)
      by each call and should be used as start for the next call. 0-0 means the scan is complete.
      
      This PR extends the deferred reply mechanism for any bulk string (not just counts)
      
      This PR carries some unrelated test code changes:
      - Renames the term "client" into "consumer" in the stream-cgroups test
      - And also changes DEBUG SLEEP into "after"
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      714e103a
  2. 04 Jan, 2021 1 commit
    • Itamar Haber's avatar
      HELP subcommand, continued (#5531) · 9dcdc7e7
      Itamar Haber authored
      
      
      * man-like consistent long formatting
      * Uppercases commands, subcommands and options
      * Adds 'HELP' to HELP for all
      * Lexicographical order
      * Uses value notation and other .md likeness
      * Moves const char *help to top
      * Keeps it under 80 chars
      * Misc help typos, consistent conjuctioning (i.e return and not returns)
      * Uses addReplySubcommandSyntaxError(c) all over
      Signed-off-by: default avatarItamar Haber <itamar@redislabs.com>
      9dcdc7e7
  3. 31 Dec, 2020 1 commit
    • filipe oliveira's avatar
      Add errorstats info section, Add failed_calls and rejected_calls to commandstats (#8217) · 90b9f08e
      filipe oliveira authored
      This Commit pushes forward the observability on overall error statistics and command statistics within redis-server:
      
      It extends INFO COMMANDSTATS to have
      - failed_calls in - so we can keep track of errors that happen from the command itself, broken by command.
      - rejected_calls - so we can keep track of errors that were triggered outside the commmand processing per se
      
      Adds a new section to INFO, named ERRORSTATS that enables keeping track of the different errors that
      occur within redis ( within processCommand and call ) based on the reply Error Prefix ( The first word
      after the "-", up to the first space ).
      
      This commit also fixes RM_ReplyWithError so that it can be correctly identified as an error reply.
      90b9f08e
  4. 27 Dec, 2020 2 commits
    • Oran Agra's avatar
      Fix memory leaks in error replies due to recent change (#8249) · 049cf8cd
      Oran Agra authored
      Recently efaf09ee started using addReplyErrorSds in place of
      addReplySds the later takes ownership of the string but the former did
      not.
      This introduced memory leaks when a script returns an error to redis,
      and also in clusterRedirectClient (two new usages of
      addReplyErrorSds which was mostly unused till now.
      
      This commit chagnes two thanks.
      1. change addReplyErrorSds to take ownership of the error string.
      2. scripting.c doesn't actually need to use addReplyErrorSds, it's a
      perfect match for addReplyErrorFormat (replaces newlines with spaces)
      049cf8cd
    • zhaozhao.zz's avatar
      Tracking: add CLIENT TRACKINGINFO subcommand (#7309) · 299f9ebf
      zhaozhao.zz authored
      
      
      Add CLIENT TRACKINGINFO subcommand
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      299f9ebf
  5. 25 Dec, 2020 1 commit
  6. 24 Dec, 2020 14 commits
  7. 22 Dec, 2020 1 commit
    • Yossi Gottlieb's avatar
      Fix crashes with io-threads-do-reads enabled. (#8230) · e7047ec2
      Yossi Gottlieb authored
      Normally IO threads should simply read data from the socket into the
      buffer and attempt to parse it.
      
      If a protocol error is detected, a reply is generated which may result
      with installing a write handler which is not thread safe. This fix
      delays that until the client is processed back in the main thread.
      
      Fixes #8220
      e7047ec2
  8. 17 Dec, 2020 1 commit
    • Qu Chen's avatar
      Not over-allocate client query buffer when reading large objects. (#5954) · 11b3325e
      Qu Chen authored
      In response to large client query buffer optimization introduced in 1898e6ce. The calculation of the amount of
      remaining bytes we need to write to the query buffer was calculated wrong, as a result we are unnecessarily
      growing the client query buffer by sdslen(c->querybuf) always. This fix corrects that behavior.
      
      Please note the previous behavior prior to the before-mentioned change was correctly calculating the remaining
      additional bytes, and this change makes that calculate to be consistent.
      
      Useful context, the argument of size `ll` starts at qb_pos (which is now the beginning of the sds), but much of it
      may have already been read from the socket, so we only need to grow the sds for the remainder of it.
      11b3325e
  9. 08 Dec, 2020 1 commit
    • Oran Agra's avatar
      Handle output buffer limits for Module blocked clients (#8141) · 48efc25f
      Oran Agra authored
      Module blocked clients cache the response in a temporary client,
      the reply list in this client would be affected by the recent fix
      in #7202, but when the reply is later copied into the real client,
      it would have bypassed all the checks for output buffer limit, which
      would have resulted in both: responding with a partial response to
      the client, and also not disconnecting it at all.
      48efc25f
  10. 07 Dec, 2020 1 commit
  11. 06 Dec, 2020 1 commit
    • Oran Agra's avatar
      Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed · c31055db
      Oran Agra authored
      The test creates keys with various encodings, DUMP them, corrupt the payload
      and RESTORES it.
      It utilizes the recently added use-exit-on-panic config to distinguish between
       asserts and segfaults.
      If the restore succeeds, it runs random commands on the key to attempt to
      trigger a crash.
      
      It runs in two modes, one with deep sanitation enabled and one without.
      In the first one we don't expect any assertions or segfaults, in the second one
      we expect assertions, but no segfaults.
      We also check for leaks and invalid reads using valgrind, and if we find them
      we print the commands that lead to that issue.
      
      Changes in the code (other than the test):
      - Replace a few NPD (null pointer deference) flows and division by zero with an
        assertion, so that it doesn't fail the test. (since we set the server to use
        `exit` rather than `abort` on assertion).
      - Fix quite a lot of flows in rdb.c that could have lead to memory leaks in
        RESTORE command (since it now responds with an error rather than panic)
      - Add a DEBUG flag for SET-SKIP-CHECKSUM-VALIDATION so that the test don't need
        to bother with faking a valid checksum
      - Remove a pile of code in serverLogObjectDebugInfo which is actually unsafe to
        run in the crash report (see comments in the code)
      - fix a missing boundary check in lzf_decompress
      
      test suite infra improvements:
      - be able to run valgrind checks before the process terminates
      - rotate log files when restarting servers
      c31055db
  12. 11 Nov, 2020 1 commit
  13. 10 Nov, 2020 1 commit
  14. 05 Nov, 2020 1 commit
    • Yossi Gottlieb's avatar
      Add RESET command. (#7982) · 1fd456f9
      Yossi Gottlieb authored
      
      
      Perform full reset of all client connection states, is if the client was
      disconnected and re-connected. This affects:
      
      * MULTI state
      * Watched keys
      * MONITOR mode
      * Pub/Sub subscription
      * ACL/Authenticated state
      * Client tracking state
      * Cluster read-only/asking state
      * RESP version (reset to 2)
      * Selected database
      * CLIENT REPLY state
      
      The response is +RESET to make it easily distinguishable from other
      responses.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarItamar Haber <itamar@redislabs.com>
      1fd456f9
  15. 28 Oct, 2020 1 commit
    • yoav-steinberg's avatar
      Add local address to CLIENT LIST, and a CLIENT KILL filter. (#7913) · 84b3c18f
      yoav-steinberg authored
      Useful when you want to know through which bind address the client connected to
      the server in case of multiple bind addresses.
      
      - Adding `laddr` field to CLIENT list showing the local (bind) address.
      - Adding `LADDR` option to CLIENT KILL to kill all the clients connected
        to a specific local address.
      - Refactoring to share code.
      84b3c18f
  16. 05 Oct, 2020 1 commit
    • Oran Agra's avatar
      memory reporting of clients argv (#7874) · bea40e6a
      Oran Agra authored
      track and report memory used by clients argv.
      this is very usaful in case clients started sending a command and didn't
      complete it. in which case the first args of the command are already
      trimmed from the query buffer.
      
      in an effort to avoid cache misses and overheads while keeping track of
      these, i avoid calling sdsZmallocSize and instead use the sdslen /
      bulk-len which can at least give some insight into the problem.
      
      This memory is now added to the total clients memory usage, as well as
      the client list.
      bea40e6a
  17. 02 Oct, 2020 1 commit
  18. 01 Oct, 2020 1 commit
  19. 27 Sep, 2020 1 commit
  20. 24 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Don't write replies if close the client ASAP (#7202) · 57709c4b
      Wang Yuan authored
      
      
      Before this commit, we would have continued to add replies to the reply buffer even if client
      output buffer limit is reached, so the used memory would keep increasing over the configured limit.
      What's more, we shouldn’t write any reply to the client if it is set 'CLIENT_CLOSE_ASAP' flag
      because that doesn't conform to its definition and we will close all clients flagged with
      'CLIENT_CLOSE_ASAP' in ‘beforeSleep’.
      
      Because of code execution order, before this, we may firstly write to part of the replies to
      the socket before disconnecting it, but in fact, we may can’t send the full replies to clients
      since OS socket buffer is limited. But this unexpected behavior makes some commands work well,
      for instance ACL DELUSER, if the client deletes the current user, we need to send reply to client
      and close the connection, but before, we close the client firstly and write the reply to reply
      buffer. secondly, we shouldn't do this despite the fact it works well in most cases.
      
      We add a flag 'CLIENT_CLOSE_AFTER_COMMAND' to mark clients, this flag means we will close the
      client after executing commands and send all entire replies, so that we can write replies to
      reply buffer during executing commands, send replies to clients, and close them later.
      
      We also fix some implicit problems. If client output buffer limit is enforced in 'multi/exec',
      all commands will be executed completely in redis and clients will not read any reply instead of
      partial replies. Even more, if the client executes 'ACL deluser' the using user in 'multi/exec',
      it will not read the replies after 'ACL deluser' just like before executing 'client kill' itself
      in 'multi/exec'.
      
      We added some tests for output buffer limit breach during multi-exec and using a pipeline of
      many small commands rather than one with big response.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      57709c4b
  21. 22 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Kill disk-based fork child when all replicas drop and 'save' is not enabled (#7819) · 1bb5794a
      Wang Yuan authored
      When all replicas waiting for a bgsave get disconnected (possibly due to output buffer limit),
      It may be good to kill the bgsave child. in diskless replication it already happens, but in
      disk-based, the child may still serve some purpose (for persistence).
      
      By killing the child, we prevent it from eating COW memory in vain, and we also allow a new child fork sooner for the next full synchronization or bgsave.
      We do that only if rdb persistence wasn't enabled in the configuration.
      
      Btw, now, rdbRemoveTempFile in killRDBChild won't block server, so we can killRDBChild safely.
      1bb5794a
  22. 21 Sep, 2020 1 commit
  23. 17 Sep, 2020 1 commit
    • Wang Yuan's avatar
      Implement redisAtomic to replace _Atomic C11 builtin (#7707) · 445a4b66
      Wang Yuan authored
      Redis 6.0 introduces I/O threads, it is so cool and efficient, we use C11
      _Atomic to establish inter-thread synchronization without mutex. But the
      compiler that must supports C11 _Atomic can compile redis code, that brings a
      lot of inconvenience since some common platforms can't support by default such
      as CentOS7, so we want to implement redis atomic type to make it more portable.
      
      We have implemented our atomic variable for redis that only has 'relaxed'
      operations in src/atomicvar.h, so we implement some operations with
      'sequentially-consistent', just like the default behavior of C11 _Atomic that
      can establish inter-thread synchronization. And we replace all uses of C11
      _Atomic with redis atomic variable.
      
      Our implementation of redis atomic variable uses C11 _Atomic, __atomic or
      __sync macros if available, it supports most common platforms, and we will
      detect automatically which feature we use. In Makefile we use a dummy file to
      detect if the compiler supports C11 _Atomic. Now for gcc, we can compile redis
      code theoretically if your gcc version is not less than 4.1.2(starts to support
      __sync_xxx operations). Otherwise, we remove use mutex fallback to implement
      redis atomic variable for performance and test. You will get compiling errors
      if your compiler doesn't support all features of above.
      
      For cover redis atomic variable tests, we add other CI jobs that build redis on
      CentOS6 and CentOS7 and workflow daily jobs that run the tests on them.
      For them, we just install gcc by default in order to cover different compiler
      versions, gcc is 4.4.7 by default installation on CentOS6 and 4.8.5 on CentOS7.
      
      We restore the feature that we can test redis with Helgrind to find data race
      errors. But you need install Valgrind in the default path configuration firstly
      before running your tests, since we use macros in helgrind.h to tell Helgrind
      inter-thread happens-before relationship explicitly for avoiding false positives.
      Please open an issue on github if you find data race errors relate to this commit.
      
      Unrelated:
      - Fix redefinition of typedef 'RedisModuleUserChangedFunc'
        For some old version compilers, they will report errors or warnings, if we
        re-define function type.
      445a4b66
  24. 16 Sep, 2020 2 commits
    • WuYunlong's avatar
      f8660233
    • Jim Brunner's avatar
      Incremental eviction processing (#7653) · 810e28a3
      Jim Brunner authored
      Rather than blindly evicting until maxmemory limit is achieved, this
      update adds a time limit to eviction.  While over the maxmemory limit,
      eviction will process before each command AND as a timeProc when no
      commands are running.
      
      This will reduce the latency impact on many cases, especially pathological
      cases like massive used memory increase during dict rehashing.
      
      There is a risk that some other edge cases (like massive pipelined use
      of MGET) could cause Redis memory usage to keep growing despite the
      eviction attempts, so a new maxmemory-eviction-tenacity config is
      introduced to let users mitigate that.
      810e28a3