1. 09 Sep, 2021 1 commit
    • sundb's avatar
      Replace all usage of ziplist with listpack for t_zset (#9366) · 3ca6972e
      sundb authored
      Part two of implementing #8702 (zset), after #8887.
      
      ## Description of the feature
      Replaced all uses of ziplist with listpack in t_zset, and optimized some of the code to optimize performance.
      
      ## Rdb format changes
      New `RDB_TYPE_ZSET_LISTPACK` rdb type.
      
      ## Rdb loading improvements:
      1) Pre-expansion of dict for validation of duplicate data for listpack and ziplist.
      2) Simplifying the release of empty key objects when RDB loading.
      3) Unify ziplist and listpack data verify methods for zset and hash, and move code to rdb.c.
      
      ## Interface changes
      1) New `zset-max-listpack-entries` config is an alias for `zset-max-ziplist-entries` (same with `zset-max-listpack-value`).
      2) OBJECT ENCODING will return listpack instead of ziplist.
      
      ## Listpack improvements:
      1) Add `lpDeleteRange` and `lpDeleteRangeWithEntry` functions to delete a range of entries from listpack.
      2) Improve the performance of `lpCompare`, converting from string to integer is faster than converting from integer to string.
      3) Replace `snprintf` with `ll2string` to improve performance in converting numbers to strings in `lpGet()`.
      
      ## Zset improvements:
      1) Improve the performance of `zzlFind` method, use `lpFind` instead of `lpCompare` in a loop.
      2) Use `lpDeleteRangeWithEntry` instead of `lpDelete` twice to delete a element of zset.
      
      ## Tests
      1) Add some unittests for `lpDeleteRange` and `lpDeleteRangeWithEntry` function.
      2) Add zset RDB loading test.
      3) Add benchmark test for `lpCompare` and `ziplsitCompare`.
      4) Add empty listpack zset corrupt dump test.
      3ca6972e
  2. 29 Aug, 2021 1 commit
  3. 23 Aug, 2021 1 commit
    • Wen Hui's avatar
      config memory limits: handle values larger than (signed) LLONG_MAX (#9313) · 641780a9
      Wen Hui authored
      This aims to solve the issue in CONFIG SET maxmemory can only set maxmemory to up
      to 9223372036854775807 (2^63) while the maxmemory should be ULLONG.
      Added a memtoull function to convert a string representing an amount of memory
      into the number of bytes (similar to memtoll but for ull). Also added ull2string to
      convert a ULLong to string (Similar to ll2string).
      641780a9
  4. 22 Aug, 2021 1 commit
  5. 10 Aug, 2021 1 commit
    • sundb's avatar
      Replace all usage of ziplist with listpack for t_hash (#8887) · 02fd76b9
      sundb authored
      
      
      Part one of implementing #8702 (taking hashes first before other types)
      
      ## Description of the feature
      1. Change ziplist encoded hash objects to listpack encoding.
      2. Convert existing ziplists on RDB loading time. an O(n) operation.
      
      ## Rdb format changes
      1. Add RDB_TYPE_HASH_LISTPACK rdb type.
      2. Bump RDB_VERSION to 10
      
      ## Interface changes
      1. New `hash-max-listpack-entries` config is an alias for `hash-max-ziplist-entries` (same with `hash-max-listpack-value`)
      2. OBJECT ENCODING will return `listpack` instead of `ziplist`
      
      ## Listpack improvements:
      1. Support direct insert, replace integer element (rather than convert back and forth from string)
      3. Add more listpack capabilities to match the ziplist ones (like `lpFind`, `lpRandomPairs` and such)
      4. Optimize element length fetching, avoid multiple calculations
      5. Use inline to avoid function call overhead.
      
      ## Tests
      1. Add a new test to the RDB load time conversion
      2. Adding the listpack unit tests. (based on the one in ziplist.c)
      3. Add a few "corrupt payload: fuzzer findings" tests, and slightly modify existing ones.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      02fd76b9
  6. 05 Aug, 2021 2 commits
    • Madelyn Olson's avatar
      Add debug config flag to print certain config values on engine crash (#9304) · 39a4a44d
      Madelyn Olson authored
      Add debug config flag to print certain config values on engine crash
      39a4a44d
    • yoav-steinberg's avatar
      dict struct memory optimizations (#9228) · 5e908a29
      yoav-steinberg authored
      Reduce dict struct memory overhead
      on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin.
      
      summary of changes:
      - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below).
      - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c)
      - Eliminate the `sizemask` field, can be calculated from size when needed.
      - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8.
      
      Interface change: pass dict pointer to dict type call back functions.
      This is instead of passing the removed privdata field. In the future if
      we'd like to have private data in the callbacks we can extract it from
      the dict type. We can extend dictType to include a custom dict struct
      allocator and use it to allocate more data at the end of the dict
      struct. This data can then be used to store private data later acccessed
      by the callbacks.
      5e908a29
  7. 02 Aug, 2021 1 commit
    • Binbin's avatar
      Modify some error logs printing level. (#9306) · 4000cb7d
      Binbin authored
      1. In sendBulkToSlave, we used LL_VERBOSE in the past, changed to
      LL_WARNING. (all the other places that do freeClient(slave) use LL_WARNING)
      2. The old style LOG_WARNING, chang it to LL_WARNING. Introduced in an
      old pr (#1690).
      4000cb7d
  8. 24 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add bind-source-addr configuration argument. (#9142) · f233c4c5
      Yossi Gottlieb authored
      In the past, the first bind address that was explicitly specified was
      also used to bind outgoing connections. This could result with some
      problems. For example: on some systems using `bind 127.0.0.1` would
      result with outgoing connections also binding to `127.0.0.1` and failing
      to connect to remote addresses.
      
      With the recent change to the way `bind` is handled, this presented
      other issues:
      
      * The default first bind address is '*' which is not a valid address.
      * We make no distinction between user-supplied config that is identical
      to the default, and the default config.
      
      This commit addresses both these issues by introducing an explicit
      configuration parameter to control the bind address on outgoing
      connections.
      f233c4c5
  9. 22 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve bind and protected-mode config handling. (#9034) · 07b0d144
      Yossi Gottlieb authored
      * Specifying an empty `bind ""` configuration prevents Redis from listening on any TCP port. Before this commit, such configuration was not accepted.
      * Using `CONFIG GET bind` will always return an explicit configuration value. Before this commit, if a bind address was not specified the returned value was empty (which was an anomaly).
      
      Another behavior change is that modifying the `bind` configuration to a non-default value will NO LONGER DISABLE protected-mode implicitly.
      07b0d144
  10. 20 Jun, 2021 3 commits
  11. 16 Jun, 2021 3 commits
    • Sam Bortman's avatar
      Support glob pattern matching for config include files (#8980) · c2b93ff8
      Sam Bortman authored
      This will allow distros to use an "include conf.d/*.conf" statement in the default configuration file
      which will facilitate customization across upgrades/downgrades.
      
      The change itself is trivial: instead of opening an individual file, the glob call creates a vector of files to open, and each file is opened in turn, and its content is added to the configuration.
      c2b93ff8
    • yoav-steinberg's avatar
      Remove gopher protocol support. (#9057) · 362786c5
      yoav-steinberg authored
      Gopher support was added mainly because it was simple (trivial to add).
      But apparently even something that was trivial at the time, does cause complications
      down the line when adding more features.
      We recently ran into a few issues with io-threads conflicting with the gopher support.
      We had to either complicate the code further in order to solve them, or drop gopher.
      AFAIK it's completely unused, so we wanna chuck it, rather than keep supporting it.
      362786c5
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  12. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  13. 01 Jun, 2021 1 commit
  14. 20 May, 2021 1 commit
  15. 19 May, 2021 1 commit
  16. 05 Apr, 2021 1 commit
    • Huang Zhw's avatar
      Fix "default" and overwritten / reset users will not have pubsub channels... · 3b74b550
      Huang Zhw authored
      Fix "default" and overwritten / reset users will not have pubsub channels permissions by default. (#8723)
      
      Background:
      Redis 6.2 added ACL control for pubsub channels (#7993), which were supposed
      to be permissive by default to retain compatibility with redis 6.0 ACL. 
      But due to a bug, only newly created users got this `acl-pubsub-default` applied,
      while overwritten (updated) users got reset to `resetchannels` (denied).
      
      Since the "default" user exists before loading the config file,
      any ACL change to it, results in an update / overwrite.
      
      So when a "default" user is loaded from config file or include ACL
      file with no channels related rules, the user will not have any
      permissions to any channels. But other users will have default
      permissions to any channels.
      
      When upgraded from 6.0 with config rewrite, this will lead to
      "default" user channels permissions lost.
      When users are loaded from include file, then call "acl load", users
      will also lost channels permissions.
      
      Similarly, the `reset` ACL rule, would have reset the user to be denied
      access to any channels, ignoring `acl-pubsub-default` and breaking
      compatibility with redis 6.0.
      
      The implication of this fix is that it regains compatibility with redis 6.0,
      but breaks compatibility with redis 6.2.0 and 2.0.1. e.g. after the upgrade,
      the default user will regain access to pubsub channels.
      
      Other changes:
      Additionally this commit rename server.acl_pubusub_default to
      server.acl_pubsub_default and fix typo in acl tests.
      3b74b550
  17. 04 Apr, 2021 1 commit
    • Sokolov Yura's avatar
      Add cluster-allow-replica-migration option. (#5285) · 1cab9620
      Sokolov Yura authored
      
      
      Previously (and by default after commit) when master loose its last slot
      (due to migration, for example), its replicas will migrate to new last slot
      holder.
      
      There are cases where this is not desired:
      * Consolidation that results with removed nodes (including the replica, eventually).
      * Manually configured cluster topologies, which the admin wishes to preserve.
      
      Needlessly migrating a replica triggers a full synchronization and can have a negative impact, so
      we prefer to be able to avoid it where possible.
      
      This commit adds 'cluster-allow-replica-migration' configuration option that is
      enabled by default to preserve existed behavior. When disabled, replicas will
      not be auto-migrated.
      
      Fixes #4896
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1cab9620
  18. 30 Mar, 2021 3 commits
    • Jérôme Loyet's avatar
      Add replica-announced config option (#8653) · 91f4f416
      Jérôme Loyet authored
      The 'sentinel replicas <master>' command will ignore replicas with
      `replica-announced` set to no.
      
      The goal of disabling the config setting replica-announced is to allow ghost
      replicas. The replica is in the cluster, synchronize with its master, can be
      promoted to master and is not exposed to sentinel clients. This way, it is
      acting as a live backup or living ghost.
      
      In addition, to prevent the replica to be promoted as master, set
      replica-priority to 0.
      91f4f416
    • Viktor Söderqvist's avatar
      Add support for plaintext clients in TLS cluster (#8587) · 5629dbe7
      Viktor Söderqvist authored
      The cluster bus is established over TLS or non-TLS depending on the configuration tls-cluster. The client ports distributed in the cluster and sent to clients are assumed to be TLS or non-TLS also depending on tls-cluster.
      
      The cluster bus is now extended to also contain the non-TLS port of clients in a TLS cluster, when available. The non-TLS port of a cluster node, when available, is sent to clients connected without TLS in responses to CLUSTER SLOTS, CLUSTER NODES, CLUSTER SLAVES and MOVED and ASK redirects, instead of the TLS port.
      
      The user was able to override the client port by defining cluster-announce-port. Now cluster-announce-tls-port is added, so the user can define an alternative announce port for both TLS and non-TLS clients.
      
      Fixes #8134
      5629dbe7
    • JunhuaY's avatar
      re-fix config rewrite for empty save directive (#8722) · 28375ff6
      JunhuaY authored
      the bug was also discussed in #8716, and was solved in #8719, but incompletely:
      when the server is started, and the save option is default, if you issue the " config set save "" "
      to change the save option, and then issue the “config rewrite” command, the " save "" " won't be saved.
      28375ff6
  19. 29 Mar, 2021 1 commit
  20. 26 Mar, 2021 1 commit
  21. 22 Mar, 2021 1 commit
  22. 16 Mar, 2021 1 commit
  23. 05 Mar, 2021 1 commit
  24. 01 Mar, 2021 1 commit
    • YaacovHazan's avatar
      Make port, tls-port and bind configurations modifiable (#8510) · a031d268
      YaacovHazan authored
      Add ability to modify port, tls-port and bind configurations by CONFIG SET command.
      
      To simplify the code and make it cleaner, a new structure
      added, socketFds, which contains the file descriptors array and its counter,
      and used for TCP, TLS and Cluster sockets file descriptors.
      a031d268
  25. 26 Feb, 2021 1 commit
  26. 22 Feb, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix integer overflow (CVE-2021-21309). (#8522) · d32f2e99
      Yossi Gottlieb authored
      On 32-bit systems, setting the proto-max-bulk-len config parameter to a high value may result with integer overflow and a subsequent heap overflow when parsing an input bulk (CVE-2021-21309).
      
      This fix has two parts:
      
      Set a reasonable limit to the config parameter.
      Add additional checks to prevent the problem in other potential but unknown code paths.
      d32f2e99
  27. 28 Jan, 2021 2 commits
    • Yossi Gottlieb's avatar
      Add proc-title-template option. (#8397) · 4bb5ccbe
      Yossi Gottlieb authored
      Make it possible to customize the process title, i.e. include custom
      strings, immutable configuration like port, tls-port, unix socket name,
      etc.
      4bb5ccbe
    • Z. Liu's avatar
      Add 'set-proc-title' config so that this mechanism can be disabled (#3623) · 17b34c73
      Z. Liu authored
      
      
      if option `set-proc-title' is no, then do nothing for proc title.
      
      The reason has been explained long ago, see following:
      
      We update redis to 2.8.8, then found there are some side effect when
      redis always change the process title.
      
      We run several slave instance on one computer, and all these salves
      listen on unix socket only, then ps will show:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0
      
      for redis 2.6 the output of ps is like following:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf
      
      Later is more informational in our case. The situation
      is worse when we manage the config and process running
      state by salt. Salt check the process by running "ps |
      grep SIG" (for Gentoo System) to check the running
      state, where SIG is the string to search for when
      looking for the service process with ps. Previously, we
      define sig as "/usr/sbin/redis-server
      /etc/redis/a.conf". Since the ps output is identical for
      our case, so we have no way to check the state of
      specified redis instance.
      
      So, for our case, we prefer the old behavior, i.e, do
      not change the process title for the main redis process.
      Or add an option such as "set-proc-title [yes|no]" to
      control this behavior.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      17b34c73
  28. 26 Jan, 2021 1 commit
    • Wen Hui's avatar
      Sentinel: Fix Config Dependency and Rewrite Sequence (#8271) · 1aad55b6
      Wen Hui authored
      This commit fixes a well known and an annoying issue in Sentinel mode.
      
      Cause of this issue:
      Currently, Redis rewrite process works well in server mode, however in sentinel mode,
      the sentinel config has variant semantics for different configurations, in example configuration
      https://github.com/redis/redis/blob/unstable/sentinel.conf, we put comments on these.
      However the rewrite process only treat the sentinel config as a single option. During rewrite
      process, it will mess up with the lines and comments.
      
      Approaches:
      In order to solve this issue, we need to differentiate different subconfig options in sentinel separately,
      for example, sentinel monitor <master-name> <ip> <redis-port> <quorum>
      we can treat it as sentinel monitor option, instead of the sentinel option.
      
      This commit also fixes the dependency issue when putting configurations in sentinel.conf.
      For example before this commit,we must put
      `sentinel monitor <master-name> <ip> <redis-port> <quorum>` before
      `sentinel auth-pass <master-name> <password>` for a single master,
      otherwise the server cannot start and will return error. This commit fixes this issue, as long as
      the monitoring master was configured, no matter the sequence is, the sentinel can start and run properly.
      1aad55b6
  29. 21 Jan, 2021 1 commit
  30. 20 Jan, 2021 1 commit
    • Yossi Gottlieb's avatar
      CONFIG REWRITE should honor umask settings. (#8371) · b548ffab
      Yossi Gottlieb authored
      Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one.
      
      The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly.
      
      Fixes #8369
      b548ffab
  31. 15 Jan, 2021 1 commit
    • Yang Bodong's avatar
      Add lazyfree-lazy-user-flush config to control default behavior of... · 294f93af
      Yang Bodong authored
      Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258)
      
      * Adds ASYNC and SYNC arguments to SCRIPT FLUSH
      * Adds SYNC argument to FLUSHDB and FLUSHALL
      * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH.
      
      the new behavior is as follows:
      * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the
        value of lazyfree-lazy-user-flush.
      * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner.
      * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
      294f93af
  32. 07 Jan, 2021 1 commit
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · b02780c4
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b02780c4