1. 18 Oct, 2021 1 commit
  2. 15 Sep, 2021 1 commit
    • guybe7's avatar
      A better approach for COMMAND INFO for movablekeys commands (#8324) · 03fcc211
      guybe7 authored
      Fix #7297
      
      The problem:
      
      Today, there is no way for a client library or app to know the key name indexes for commands such as
      ZUNIONSTORE/EVAL and others with "numkeys", since COMMAND INFO returns no useful info for them.
      
      For cluster-aware redis clients, this requires to 'patch' the client library code specifically for each of these commands or to
      resolve each execution of these commands with COMMAND GETKEYS.
      
      The solution:
      
      Introducing key specs other than the legacy "range" (first,last,step)
      
      The 8th element of the command info array, if exists, holds an array of key specs. The array may be empty, which indicates
      the command doesn't take any key arguments or may contain one or more key-specs, each one may leads to the discovery
      of 0 or more key arguments.
      
      A client library that doesn't support this key-spec feature will keep using the first,last,step and movablekeys flag which will
      obviously remain unchanged.
      
      A client that supports this key-specs feature needs only to look at the key-specs array. If it finds an unrecognized spec, it
      must resort to using COMMAND GETKEYS if it wishes to get all key name arguments, but if all it needs is one key in order
      to know which cluster node to use, then maybe another spec (if the command has several) can supply that, and there's no
      need to use GETKEYS.
      
      Each spec is an array of arguments, first one is the spec name, the second is an array of flags, and the third is an array
      containing details about the spec (specific meaning for each spec type)
      The initial flags we support are "read" and "write" indicating if the keys that this key-spec finds are used for read or for write.
      clients should ignore any unfamiliar flags.
      
      In order to easily find the positions of keys in a given array of args we introduce keys specs. There are two logical steps of
      key specs:
      1. `start_search`: Given an array of args, indicate where we should start searching for keys
      2. `find_keys`: Given the output of start_search and an array of args, indicate all possible indices of keys.
      
      ### start_search step specs
      - `index`: specify an argument index explicitly
        - `index`: 0 based index (1 means the first command argument)
      - `keyword`: specify a string to match in `argv`. We should start searching for keys just after the keyword appears.
        - `keyword`: the string to search for
        - `start_search`: an index from which to start the keyword search (can be negative, which means to search from the end)
      
      Examples:
      - `SET` has start_search of type `index` with value `1`
      - `XREAD` has start_search of type `keyword` with value `[“STREAMS”,1]`
      - `MIGRATE` has start_search of type `keyword` with value `[“KEYS”,-2]`
      
      ### find_keys step specs
      - `range`: specify `[count, step, limit]`.
        - `lastkey`: index of the last key. relative to the index returned from begin_search. -1 indicating till the last argument, -2 one before the last
        - `step`: how many args should we skip after finding a key, in order to find the next one
        - `limit`: if count is -1, we use limit to stop the search by a factor. 0 and 1 mean no limit. 2 means ½ of the remaining args, 3 means ⅓, and so on.
      - “keynum”: specify `[keynum_index, first_key_index, step]`.
        - `keynum_index`: is relative to the return of the `start_search` spec.
        - `first_key_index`: is relative to `keynum_index`.
        - `step`: how many args should we skip after finding a key, in order to find the next one
      
      Examples:
      - `SET` has `range` of `[0,1,0]`
      - `MSET` has `range` of `[-1,2,0]`
      - `XREAD` has `range` of `[-1,1,2]`
      - `ZUNION` has `start_search` of type `index` with value `1` and `find_keys` of type `keynum` with value `[0,1,1]`
      - `AI.DAGRUN` has `start_search` of type `keyword` with value `[“LOAD“,1]` and `find_keys` of type `keynum` with value
        `[0,1,1]` (see https://oss.redislabs.com/redisai/master/commands/#aidagrun)
      
      Note: this solution is not perfect as the module writers can come up with anything, but at least we will be able to find the key
      args of the vast majority of commands.
      If one of the above specs can’t describe the key positions, the module writer can always fall back to the `getkeys-api` option.
      
      Some keys cannot be found easily (`KEYS` in `MIGRATE`: Imagine the argument for `AUTH` is the string “KEYS” - we will
      start searching in the wrong index). 
      The guarantee is that the specs may be incomplete (`incomplete` will be specified in the spec to denote that) but we never
      report false information (assuming the command syntax is correct).
      For `MIGRATE` we start searching from the end - `startfrom=-1` - and if one of the keys is actually called "keys" we will
      report only a subset of all keys - hence the `incomplete` flag.
      Some `incomplete` specs can be completely empty (i.e. UNKNOWN begin_search) which should tell the client that
      COMMAND GETKEYS (or any other way to get the keys) must be used (Example: For `SORT` there is no way to describe
      the STORE keyword spec, as the word "store" can appear anywhere in the command).
      
      We will expose these key specs in the `COMMAND` command so that clients can learn, on startup, where the keys are for
      all commands instead of holding hardcoded tables or use `COMMAND GETKEYS` in runtime.
      
      Comments:
      1. Redis doesn't internally use the new specs, they are only used for COMMAND output.
      2. In order to support the current COMMAND INFO format (reply array indices 4, 5, 6) we created a synthetic range, called
         legacy_range, that, if possible, is built according to the new specs.
      3. Redis currently uses only getkeys_proc or the legacy_range to get the keys indices (in COMMAND GETKEYS for
         example).
      
      "incomplete" specs:
      the command we have issues with are MIGRATE, STRALGO, and SORT
      for MIGRATE, because the token KEYS, if exists, must be the last token, we can search in reverse. it one of the keys is
      actually the string "keys" will return just a subset of the keys (hence, it's "incomplete")
      for SORT and STRALGO we can use this heuristic (the keys can be anywhere in the command) and therefore we added a
      key spec that is both "incomplete" and of "unknown type"
      
      if a client encounters an "incomplete" spec it means that it must find a different way (either COMMAND GETKEYS or have
      its own parser) to retrieve the keys.
      please note that all commands, apart from the three mentioned above, have "complete" key specs
      03fcc211
  3. 05 Aug, 2021 2 commits
    • Wen Hui's avatar
      Add sentinel debug option command (#9291) · 63e2a6d2
      Wen Hui authored
      
      
      This makes it possible to tune many parameters that were previously hard coded.
      We don't intend these to be user configurable, but only used by tests to accelerate certain conditions which would otherwise take a long time and slow down the test suite.
      Co-authored-by: default avatarLucas Guang Yang <l84193800@china.huawei.com>
      63e2a6d2
    • yoav-steinberg's avatar
      dict struct memory optimizations (#9228) · 5e908a29
      yoav-steinberg authored
      Reduce dict struct memory overhead
      on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin.
      
      summary of changes:
      - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below).
      - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c)
      - Eliminate the `sizemask` field, can be calculated from size when needed.
      - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8.
      
      Interface change: pass dict pointer to dict type call back functions.
      This is instead of passing the removed privdata field. In the future if
      we'd like to have private data in the callbacks we can extract it from
      the dict type. We can extend dictType to include a custom dict struct
      allocator and use it to allocate more data at the end of the dict
      struct. This data can then be used to store private data later acccessed
      by the callbacks.
      5e908a29
  4. 29 Jul, 2021 1 commit
    • Wen Hui's avatar
      Remove duplicate zero-port sentinels (#9240) · db415364
      Wen Hui authored
      The issue is that when a sentinel with the same address and IP is turned on with a different runid, its port is set to 0 but it is still present in the dictionary master->sentinels which contain all the sentinels for a master.
      
      This causes a problem when we do INFO SENTINEL because it takes the size of the dictionary of sentinels. This might also cause a problem for failover if enough sentinels have their port set to 0 since the number of voters in failover is also determined by the size of the dictionary of sentinels.
      
      This commits removes the sentinels with the port set to zero from the dictionary of sentinels.
      Fixes #8786
      db415364
  5. 24 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add bind-source-addr configuration argument. (#9142) · f233c4c5
      Yossi Gottlieb authored
      In the past, the first bind address that was explicitly specified was
      also used to bind outgoing connections. This could result with some
      problems. For example: on some systems using `bind 127.0.0.1` would
      result with outgoing connections also binding to `127.0.0.1` and failing
      to connect to remote addresses.
      
      With the recent change to the way `bind` is handled, this presented
      other issues:
      
      * The default first bind address is '*' which is not a valid address.
      * We make no distinction between user-supplied config that is identical
      to the default, and the default config.
      
      This commit addresses both these issues by introducing an explicit
      configuration parameter to control the bind address on outgoing
      connections.
      f233c4c5
  6. 22 Jun, 2021 1 commit
  7. 16 Jun, 2021 1 commit
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  8. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  9. 04 Jun, 2021 1 commit
  10. 24 May, 2021 1 commit
  11. 23 May, 2021 1 commit
  12. 19 May, 2021 1 commit
  13. 09 May, 2021 1 commit
  14. 03 May, 2021 1 commit
  15. 25 Apr, 2021 1 commit
  16. 24 Apr, 2021 1 commit
  17. 15 Apr, 2021 1 commit
  18. 13 Apr, 2021 1 commit
  19. 01 Apr, 2021 1 commit
    • Wen Hui's avatar
      generalize config file check for sentinel (#8730) · d5935bb0
      Wen Hui authored
      The implications of this change is just that in the past when a config file was missing,
      in some cases it was exiting before printing the sever startup prints and sometimes after,
      and now it'll always exit before printing them.
      d5935bb0
  20. 30 Mar, 2021 1 commit
    • Jérôme Loyet's avatar
      Add replica-announced config option (#8653) · 91f4f416
      Jérôme Loyet authored
      The 'sentinel replicas <master>' command will ignore replicas with
      `replica-announced` set to no.
      
      The goal of disabling the config setting replica-announced is to allow ghost
      replicas. The replica is in the cluster, synchronize with its master, can be
      promoted to master and is not exposed to sentinel clients. This way, it is
      acting as a live backup or living ghost.
      
      In addition, to prevent the replica to be promoted as master, set
      replica-priority to 0.
      91f4f416
  21. 24 Mar, 2021 1 commit
  22. 23 Mar, 2021 1 commit
  23. 22 Mar, 2021 2 commits
  24. 16 Mar, 2021 1 commit
  25. 11 Mar, 2021 1 commit
  26. 05 Mar, 2021 1 commit
  27. 03 Mar, 2021 2 commits
  28. 11 Feb, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix duplicate replicas issue. (#8481) · 94bc26e6
      Yossi Gottlieb authored
      We need to store replicas referenced by their announced address (IP or
      address). Before that, if hostnames were used and the IP address
      changed, duplicate entries would have been created.
      94bc26e6
  29. 10 Feb, 2021 1 commit
  30. 28 Jan, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add hostname support in Sentinel. (#8282) · bb7cd974
      Yossi Gottlieb authored
      
      
      This is both a bugfix and an enhancement.
      
      Internally, Sentinel relies entirely on IP addresses to identify
      instances. When configured with a new master, it also requires users to
      specify and IP and not hostname.
      
      However, replicas may use the replica-announce-ip configuration to
      announce a hostname. When that happens, Sentinel fails to match the
      announced hostname with the expected IP and considers that a different
      instance, triggering reconfiguration, etc.
      
      Another use case is where TLS is used and clients are expected to match
      the hostname to connect to with the certificate's SAN attribute. To
      properly implement this configuration, it is necessary for Sentinel to
      redirect clients to a hostname rather than an IP address.
      
      The new 'resolve-hostnames' configuration parameter determines if
      Sentinel is willing to accept hostnames. It is set by default to no,
      which maintains backwards compatibility and avoids unexpected DNS
      resolution delays on systems with DNS configuration issues.
      
      Internally, Sentinel continues to identify instances by their resolved
      IP address and will also report the IP by default. The new
      'announce-hostnames' parameter determines if Sentinel should prefer to
      announce a hostname, when available, rather than an IP address. This
      applies to addresses returned to clients, as well as their
      representation in the configuration file, REPLICAOF configuration
      commands, etc.
      
      This commit also introduces SENTINEL CONFIG GET and SENTINEL CONFIG SET
      which can be used to introspect or configure global Sentinel
      configuration that was previously was only possible by directly
      accessing the configuration file and possibly restarting the instance.
      Co-authored-by: default avatarmyl1024 <myl92916@qq.com>
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      bb7cd974
  31. 26 Jan, 2021 1 commit
    • Wen Hui's avatar
      Sentinel: Fix Config Dependency and Rewrite Sequence (#8271) · 1aad55b6
      Wen Hui authored
      This commit fixes a well known and an annoying issue in Sentinel mode.
      
      Cause of this issue:
      Currently, Redis rewrite process works well in server mode, however in sentinel mode,
      the sentinel config has variant semantics for different configurations, in example configuration
      https://github.com/redis/redis/blob/unstable/sentinel.conf, we put comments on these.
      However the rewrite process only treat the sentinel config as a single option. During rewrite
      process, it will mess up with the lines and comments.
      
      Approaches:
      In order to solve this issue, we need to differentiate different subconfig options in sentinel separately,
      for example, sentinel monitor <master-name> <ip> <redis-port> <quorum>
      we can treat it as sentinel monitor option, instead of the sentinel option.
      
      This commit also fixes the dependency issue when putting configurations in sentinel.conf.
      For example before this commit,we must put
      `sentinel monitor <master-name> <ip> <redis-port> <quorum>` before
      `sentinel auth-pass <master-name> <password>` for a single master,
      otherwise the server cannot start and will return error. This commit fixes this issue, as long as
      the monitoring master was configured, no matter the sequence is, the sentinel can start and run properly.
      1aad55b6
  32. 21 Jan, 2021 2 commits
  33. 19 Jan, 2021 1 commit
  34. 17 Jan, 2021 1 commit
  35. 14 Jan, 2021 1 commit
  36. 04 Jan, 2021 1 commit
    • Itamar Haber's avatar
      HELP subcommand, continued (#5531) · 9dcdc7e7
      Itamar Haber authored
      
      
      * man-like consistent long formatting
      * Uppercases commands, subcommands and options
      * Adds 'HELP' to HELP for all
      * Lexicographical order
      * Uses value notation and other .md likeness
      * Moves const char *help to top
      * Keeps it under 80 chars
      * Misc help typos, consistent conjuctioning (i.e return and not returns)
      * Uses addReplySubcommandSyntaxError(c) all over
      Signed-off-by: default avatarItamar Haber <itamar@redislabs.com>
      9dcdc7e7