1. 04 Jan, 2022 2 commits
    • yoav-steinberg's avatar
      `redis-cli --replica` reads dummy empty rdb instead of full snapshot (#10044) · 65a76357
      yoav-steinberg authored
      This makes redis-cli --replica much faster and reduces COW/fork risks on server side.
      This commit also improves the RDB filtering via REPLCONF rdb-filter-only to support no "include" specifiers at all.
      65a76357
    • Binbin's avatar
      Print error messages in monitor/pubsub when errors occurs (#10050) · c57e41c0
      Binbin authored
      In monitor/pubsub mode, if the server closes the connection,
      for example, use `CLIENT KILL`, redis-cli will exit directly
      without printing any error messages.
      
      This commit ensures that redis-cli will try to print the
      error messages before exiting. Also there is a minor cleanup
      for restart, see the example below.
      
      before:
      ```
      127.0.0.1:6379> monitor
      OK
      [root@ redis]#
      
      127.0.0.1:6379> subscribe channel
      Reading messages... (press Ctrl-C to quit)
      1) "subscribe"
      2) "channel"
      3) (integer) 1
      [root@ redis]#
      
      127.0.0.1:6379> restart
      127.0.0.1:6379> get keyUse 'restart' only in Lua debugging mode.
      (nil)
      ```
      
      after:
      ```
      127.0.0.1:6379> monitor
      OK
      Error: Server closed the connection
      [root@ redis]#
      
      127.0.0.1:6379> subscribe channel
      Reading messages... (press Ctrl-C to quit)
      1) "subscribe"
      2) "channel"
      3) (integer) 1
      Error: Server closed the connection
      [root@ redis]#
      
      127.0.0.1:6379> restart
      Use 'restart' only in Lua debugging mode.
      ```
      c57e41c0
  2. 03 Jan, 2022 1 commit
  3. 02 Jan, 2022 1 commit
    • yoav-steinberg's avatar
      Generate RDB with Functions only via redis-cli --functions-rdb (#9968) · 1bf6d6f1
      yoav-steinberg authored
      
      
      This is needed in order to ease the deployment of functions for ephemeral cases, where user
      needs to spin up a server with functions pre-loaded.
      
      #### Details:
      
      * Added `--functions-rdb` option to _redis-cli_.
      * Functions only rdb via `REPLCONF rdb-filter-only functions`. This is a placeholder for a space
        separated inclusion filter for the RDB. In the future can be `REPLCONF rdb-filter-only
        "functions db:3 key-patten:user*"` and a complementing `rdb-filter-exclude` `REPLCONF`
        can also be added.
      * Handle "slave requirements" specification to RDB saving code so we can use the same RDB
        when different slaves express the same requirements (like functions-only) and not share the
        RDB when their requirements differ. This is currently just a flags `int`, but can be extended to
        a more complex structure with various filter fields.
      * make sure to support filters only in diskless replication mode (not to override the persistence file),
        we do that by forcing diskless (even if disabled by config)
      
      other changes:
      * some refactoring in rdb.c (extract portion of a big function to a sub-function)
      * rdb_key_save_delay used in AOFRW too
      * sendChildInfo takes the number of updated keys (incremental, rather than absolute)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1bf6d6f1
  4. 30 Dec, 2021 1 commit
    • Binbin's avatar
      redis-cli: Add -X option and extend --cluster call take arg from stdin (#9980) · 4836ae32
      Binbin authored
      There are two changes in this commit:
      
      1. Add -X option to redis-cli.
      Currently `-x` can only be used to provide the last argument,
      so you can do `redis-cli dump keyname > key.dump`,
      and then do `redis-cli -x restore keyname 0 < key.dump`.
      
      But what if you want to add the replace argument (which comes last?).
      oran suggested adding such usage:
      `redis-cli -X <tag> restore keyname <tag> replace < key.dump`
      
      i.e. you're able to provide a string in the arguments that's gonna be
      substituted with the content from stdin.
      
      Note that the tag name should not conflict with others non-replaced args.
      And the -x and -X options are conflicting.
      
      Some usages:
      ```
      [root]# echo mypasswd | src/redis-cli -X passwd_tag mset username myname password passwd_tag                                                   OK
      [root]# echo username > username.txt
      [root]# head -c -1 username.txt | src/redis-cli -X name_tag mget name_tag password
      1) "myname"
      2) "mypasswd\n"
      ```
      
      2. Handle the combination of both `-x` and `--cluster` or `-X` and `--cluster`
      Extend the broadcast option to receive the last arg or <tag> arg from the stdin.
      
      Now we can use `redis-cli -x --cluster call <host>:<port> cmd`,
      or `redis-cli -X <tag> --cluster call <host>:<port> cmd <tag>`.
      (support part of #9899)
      4836ae32
  5. 26 Dec, 2021 1 commit
    • Meir Shpilraien (Spielrein)'s avatar
      Add FUNCTION DUMP and RESTORE. (#9938) · 365cbf46
      Meir Shpilraien (Spielrein) authored
      Follow the conclusions to support Functions in redis cluster (#9899)
      
      Added 2 new FUNCTION sub-commands:
      1. `FUNCTION DUMP` - dump a binary payload representation of all the functions.
      2. `FUNCTION RESTORE <PAYLOAD> [FLUSH|APPEND|REPLACE]` - give the binary payload extracted
         using `FUNCTION DUMP`, restore all the functions on the given payload. Restore policy can be given to
         control how to handle existing functions (default is APPEND):
         * FLUSH: delete all existing functions.
         * APPEND: appends the restored functions to the existing functions. On collision, abort.
         * REPLACE: appends the restored functions to the existing functions. On collision,
           replace the old function with the new function.
      
      Modify `redis-cli --cluster add-node` to use `FUNCTION DUMP` to get existing functions from
      one of the nodes in the cluster, and `FUNCTION RESTORE` to load the same set of functions
      to the new node. `redis-cli` will execute this step before sending the `CLUSTER MEET` command
      to the new node. If `FUNCTION DUMP` returns an error, assume the current Redis version do not
      support functions and skip `FUNCTION RESTORE`. If `FUNCTION RESTORE` fails, abort and do not send
      the `CLUSTER MEET` command. If the new node already contains functions (before the `FUNCTION RESTORE`
      is sent), abort and do not add the node to the cluster. Test was added to verify
      `redis-cli --cluster add-node` works as expected. 
      365cbf46
  6. 23 Dec, 2021 1 commit
    • Yuta Hongo's avatar
      redis-cli: Add OUTPUT_JSON format type (#9954) · 63f606d3
      Yuta Hongo authored
      Introduce `redis-cli --json` option.
      CSV doesn't support Map type, then parsing SLOWLOG or HMSET with multiple args are not helpful.
      
      By default the `--json` implies RESP3, which makes it much more useful, and a `-2`
      option was added to force RESP2.
      When `HELLO 3` fails, it prints a warning message (which can be silenced with `-2`).
      If a user passed `-3` explicitly, the non-interactive mode will also exit with error without
      running the command, while in interactive session it'll keep running after printing the warning.
      
      JSON output would be helpful to parse Redis replies with other tools like jq.
      
      ```
      redis-cli --json slowlog get | jq
      
      [
        [
          1,
          1639677545,
          322362,
          [
            "HMSET",
            "dummy-key",
            "field1",
            "123,456,789... (152 more bytes)",
            "field2",
            "111,222,333... (140 more bytes)",
            "field3",
            "... (349 more arguments)"
          ],
          "127.0.0.1:49312",
          ""
        ]
      ]
      
      ```
      63f606d3
  7. 19 Dec, 2021 1 commit
    • Binbin's avatar
      Fix some nonsense came from LGTM (#9962) · 0fb1aa06
      Binbin authored
      1. Local variable 's' hides a parameter of the same name.
      ```
      int anetTcpAccept(char *err, int s, char *ip, size_t ip_len, int *port) {
          if ((fd = anetGenericAccept(err,s,(struct sockaddr*)&sa,&salen)) == ANET_ERR){}
      }
      ```
      Change the parameter name from `s` to `serversock`,
      also unified with the header file definition.
      
      2. Comparison is always false because i <= 48.
      ```
      for (i = 0; i < DICT_STATS_VECTLEN-1; i++) {  // i < 49
          (i == DICT_STATS_VECTLEN-1)?">= ":"",  // i == 49
      }
      ```
      `i == DICT_STATS_VECTLEN-1` always result false, it is a dead code.
      
      3. Empty block without comment.
      `else if (!strcasecmp(opt,"ch")) {}`, add a comment to avoid warnings.
      
      4. Bit field fill of type int should have explicitly unsigned integral, explicitly signed integral, or enumeration type.
      Modify `int fill: QL_FILL_BITS;` to `unsigned int fill: QL_FILL_BITS;`
      
      5. The result of this call to reconnectingRedisCommand is not checked for null, but 80% of calls to reconnectingRedisCommand check for null.
      Just a cleanup job like others.
      0fb1aa06
  8. 02 Dec, 2021 1 commit
    • Binbin's avatar
      Fix a harmless bug when using monitor in redis-cli with wrong reply (#9875) · e3c0ea1c
      Binbin authored
      When we use monitor in redis-cli but encounter an error reply,
      we will get stuck until we press Ctrl-C to quit.
      
      This is a harmless bug. It might be useful if we add parameters
      to monitor in the future, suck as monitoring only selected db.
      
      before:
      ```
      127.0.0.1:6379> monitor wrong
      (error) ERR wrong number of arguments for 'monitor' command or subcommand
      ^C(9.98s)
      127.0.0.1:6379>
      ```
      
      after:
      ```
      127.0.0.1:6379> monitor wrong
      (error) ERR wrong number of arguments for 'monitor' command or subcommand
      127.0.0.1:6379>
      ```
      e3c0ea1c
  9. 13 Oct, 2021 1 commit
  10. 12 Oct, 2021 1 commit
  11. 23 Sep, 2021 1 commit
    • yoav-steinberg's avatar
      Client eviction (#8687) · 2753429c
      yoav-steinberg authored
      
      
      ### Description
      A mechanism for disconnecting clients when the sum of all connected clients is above a
      configured limit. This prevents eviction or OOM caused by accumulated used memory
      between all clients. It's a complimentary mechanism to the `client-output-buffer-limit`
      mechanism which takes into account not only a single client and not only output buffers
      but rather all memory used by all clients.
      
      #### Design
      The general design is as following:
      * We track memory usage of each client, taking into account all memory used by the
        client (query buffer, output buffer, parsed arguments, etc...). This is kept up to date
        after reading from the socket, after processing commands and after writing to the socket.
      * Based on the used memory we sort all clients into buckets. Each bucket contains all
        clients using up up to x2 memory of the clients in the bucket below it. For example up
        to 1m clients, up to 2m clients, up to 4m clients, ...
      * Before processing a command and before sleep we check if we're over the configured
        limit. If we are we start disconnecting clients from larger buckets downwards until we're
        under the limit.
      
      #### Config
      `maxmemory-clients` max memory all clients are allowed to consume, above this threshold
      we disconnect clients.
      This config can either be set to 0 (meaning no limit), a size in bytes (possibly with MB/GB
      suffix), or as a percentage of `maxmemory` by using the `%` suffix (e.g. setting it to `10%`
      would mean 10% of `maxmemory`).
      
      #### Important code changes
      * During the development I encountered yet more situations where our io-threads access
        global vars. And needed to fix them. I also had to handle keeps the clients sorted into the
        memory buckets (which are global) while their memory usage changes in the io-thread.
        To achieve this I decided to simplify how we check if we're in an io-thread and make it
        much more explicit. I removed the `CLIENT_PENDING_READ` flag used for checking
        if the client is in an io-thread (it wasn't used for anything else) and just used the global
        `io_threads_op` variable the same way to check during writes.
      * I optimized the cleanup of the client from the `clients_pending_read` list on client freeing.
        We now store a pointer in the `client` struct to this list so we don't need to search in it
        (`pending_read_list_node`).
      * Added `evicted_clients` stat to `INFO` command.
      * Added `CLIENT NO-EVICT ON|OFF` sub command to exclude a specific client from the
        client eviction mechanism. Added corrosponding 'e' flag in the client info string.
      * Added `multi-mem` field in the client info string to show how much memory is used up
        by buffered multi commands.
      * Client `tot-mem` now accounts for buffered multi-commands, pubsub patterns and
        channels (partially), tracking prefixes (partially).
      * CLIENT_CLOSE_ASAP flag is now handled in a new `beforeNextClient()` function so
        clients will be disconnected between processing different clients and not only before sleep.
        This new function can be used in the future for work we want to do outside the command
        processing loop but don't want to wait for all clients to be processed before we get to it.
        Specifically I wanted to handle output-buffer-limit related closing before we process client
        eviction in case the two race with each other.
      * Added a `DEBUG CLIENT-EVICTION` command to print out info about the client eviction
        buckets.
      * Each client now holds a pointer to the client eviction memory usage bucket it belongs to
        and listNode to itself in that bucket for quick removal.
      * Global `io_threads_op` variable now can contain a `IO_THREADS_OP_IDLE` value
        indicating no io-threading is currently being executed.
      * In order to track memory used by each clients in real-time we can't rely on updating
        these stats in `clientsCron()` alone anymore. So now I call `updateClientMemUsage()`
        (used to be `clientsCronTrackClientsMemUsage()`) after command processing, after
        writing data to pubsub clients, after writing the output buffer and after reading from the
        socket (and maybe other places too). The function is written to be fast.
      * Clients are evicted if needed (with appropriate log line) in `beforeSleep()` and before
        processing a command (before performing oom-checks and key-eviction).
      * All clients memory usage buckets are grouped as follows:
        * All clients using less than 64k.
        * 64K..128K
        * 128K..256K
        * ...
        * 2G..4G
        * All clients using 4g and up.
      * Added client-eviction.tcl with a bunch of tests for the new mechanism.
      * Extended maxmemory.tcl to test the interaction between maxmemory and
        maxmemory-clients settings.
      * Added an option to flag a numeric configuration variable as a "percent", this means that
        if we encounter a '%' after the number in the config file (or config set command) we
        consider it as valid. Such a number is store internally as a negative value. This way an
        integer value can be interpreted as either a percent (negative) or absolute value (positive).
        This is useful for example if some numeric configuration can optionally be set to a percentage
        of something else.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      2753429c
  12. 14 Sep, 2021 1 commit
  13. 22 Aug, 2021 1 commit
  14. 10 Aug, 2021 1 commit
  15. 05 Aug, 2021 1 commit
    • yoav-steinberg's avatar
      dict struct memory optimizations (#9228) · 5e908a29
      yoav-steinberg authored
      Reduce dict struct memory overhead
      on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin.
      
      summary of changes:
      - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below).
      - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c)
      - Eliminate the `sizemask` field, can be calculated from size when needed.
      - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8.
      
      Interface change: pass dict pointer to dict type call back functions.
      This is instead of passing the removed privdata field. In the future if
      we'd like to have private data in the callbacks we can extract it from
      the dict type. We can extend dictType to include a custom dict struct
      allocator and use it to allocate more data at the end of the dict
      struct. This data can then be used to store private data later acccessed
      by the callbacks.
      5e908a29
  16. 03 Aug, 2021 1 commit
  17. 02 Aug, 2021 1 commit
    • Huang Zhw's avatar
      When redis-cli received ASK, it didn't handle it (#8930) · cf61ad14
      Huang Zhw authored
      
      
      When redis-cli received ASK, it used string matching wrong and didn't
      handle it. 
      
      When we access a slot which is in migrating state, it maybe
      return ASK. After redirect to the new node, we need send ASKING
      command before retry the command.  In this PR after redis-cli receives 
      ASK, we send a ASKING command before send the origin command 
      after reconnecting.
      
      Other changes:
      * Make redis-cli -u and -c (unix socket and cluster mode) incompatible 
        with one another.
      * When send command fails, we avoid the 2nd reconnect retry and just
        print the error info. Users will decide how to do next. 
        See #9277.
      * Add a test faking two redis nodes in TCL to just send ASK and OK in 
        redis protocol to test ASK behavior. 
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      cf61ad14
  18. 15 Jul, 2021 1 commit
  19. 07 Jul, 2021 1 commit
    • Mikhail Fesenko's avatar
      Direct redis-cli repl prints to stderr, because --rdb can print to stdout.... · 1eb4baa5
      Mikhail Fesenko authored
      
      Direct redis-cli repl prints to stderr, because --rdb can print to stdout. fflush stdout after responses  (#9136)
      
      1. redis-cli can output --rdb data to stdout
         but redis-cli also write some messages to stdout which will mess up the rdb.
      
      2. Make redis-cli flush stdout when printing a reply
        This was needed in order to fix a hung in redis-cli test that uses
        --replica.
         Note that printf does flush when there's a newline, but fwrite does not.
      
      3. fix the redis-cli --replica test which used to pass previously
         because it didn't really care what it read, and because redis-cli
         used printf to print these other things to stdout.
      
      4. improve redis-cli --replica test to run with both diskless and disk-based.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarViktor Söderqvist <viktor@zuiderkwast.se>
      1eb4baa5
  20. 05 Jul, 2021 1 commit
  21. 30 Jun, 2021 2 commits
  22. 21 Jun, 2021 1 commit
  23. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  24. 08 Jun, 2021 2 commits
    • ikeberlein's avatar
      Make redis-cli grep friendly in pubsub mode (#9013) · 77d44bb7
      ikeberlein authored
      redis-cli is grep friendly for all commands but SUBSCRIBE/PSUBSCRIBE.
      it is unable to process output from these commands line by line piped
      to another program because of output buffering. to overcome this
      situation I propose to flush stdout each time when it is written with
      reply from these commands the same way it is already done for all other
      commands.
      77d44bb7
    • Huang Zhw's avatar
      Fix some minor bugs in redis-cli (#8982) · 1df8c129
      Huang Zhw authored
      * clusterManagerAddSlots check argv_idx error.
      
      * clusterManagerLoadInfoFromNode remove unused param opts.
      
      * redis-cli node->ip may be an sds or a c string. Using %s to format
      is always right, %S may be wrong.
      
      * In clusterManagerFixOpenSlot clusterManagerBumpEpoch call is redundant,
      because it is already called in clusterManagerSetSlotOwner.
      
      * redis-cli cluster help add more commands in help messages.
      1df8c129
  25. 02 Jun, 2021 1 commit
  26. 19 May, 2021 1 commit
  27. 18 May, 2021 2 commits
  28. 17 May, 2021 1 commit
    • Huang Zhw's avatar
      redis-cli: fix bugs in hints of commands with subcommands. (#8914) · 8827aae8
      Huang Zhw authored
      There are two bugs in redis-cli hints:
      * The hints of commands with subcommands lack first params.
      * When search matching command of currently input, we should find the
      command with longest matching prefix. If not COMMAND INFO will always
      match COMMAND and display no hints.
      8827aae8
  29. 06 May, 2021 1 commit
    • Huang Zhw's avatar
      redis-cli when SELECT fails, we should reset dbnum to 0 (#8898) · 6b475989
      Huang Zhw authored
      
      
      when SELECT fails, we should reset dbnum to 0, so the prompt will not
      display incorrectly.
      
      Additionally when SELECT and HELLO fail, we output message to inform
      it.
      
      Add config.input_dbnum which means the dbnum about to select.
      And config.dbnum means currently selected dbnum. When users succeed to
      select db, config.dbnum and config.input_dbnum will be the same. When
      users select db failed, config.input_dbnum will be kept. Next time if users
      auth success, config.input_dbnum will be automatically selected.
      When reconnect, we should select the origin dbnum.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      6b475989
  30. 04 May, 2021 1 commit
  31. 03 May, 2021 1 commit
  32. 29 Apr, 2021 1 commit
  33. 28 Apr, 2021 1 commit
  34. 26 Apr, 2021 1 commit
    • Andy Pan's avatar
      Fail fast when systemic error occurs in poll (#8749) · a8b6596d
      Andy Pan authored
      Most of the ae.c backends didn't explicitly handle errors, and instead
      ignored all errors and did an implicit retry.
      This is desired for EAGAIN and EINTER, but in case of other systematic
      errors, we prefer to fail and log the error we got rather than get into a busy loop.
      a8b6596d
  35. 21 Apr, 2021 1 commit
  36. 05 Apr, 2021 1 commit