1. 26 Jul, 2021 1 commit
    • Huang Zhw's avatar
      Add INFO stat total_eviction_exceeded_time and current_eviction_exceeded_time (#9031) · 17511df5
      Huang Zhw authored
      
      
      Add two INFO metrics:
      ```
      total_eviction_exceeded_time:69734
      current_eviction_exceeded_time:10230
      ```
      `current_eviction_exceeded_time` if greater than 0, means how much time current used memory is greater than `maxmemory`. And we are still over the maxmemory. If used memory is below `maxmemory`, this metric is reset to 0.
      `total_eviction_exceeded_time` means total time used memory is greater than `maxmemory` since server startup. 
      The units of these two metrics are ms.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      17511df5
  2. 21 Jul, 2021 1 commit
    • Huang Zhw's avatar
      On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS... · 71d45287
      Huang Zhw authored
      On 32 bit platform, the bit position of GETBIT/SETBIT/BITFIELD/BITCOUNT,BITPOS may overflow (see CVE-2021-32761) (#9191)
      
      GETBIT, SETBIT may access wrong address because of wrap.
      BITCOUNT and BITPOS may return wrapped results.
      BITFIELD may access the wrong address but also allocate insufficient memory and segfault (see CVE-2021-32761).
      
      This commit uses `uint64_t` or `long long` instead of `size_t`.
      related https://github.com/redis/redis/pull/8096
      
      At 32bit platform:
      > setbit bit 4294967295 1
      (integer) 0
      > config set proto-max-bulk-len 536870913
      OK
      > append bit "\xFF"
      (integer) 536870913
      > getbit bit 4294967296
      (integer) 0
      
      When the bit index is larger than 4294967295, size_t can't hold bit index. In the past,  `proto-max-bulk-len` is limit to 536870912, so there is no problem.
      
      After this commit, bit position is stored in `uint64_t` or `long long`. So when `proto-max-bulk-len > 536870912`, 32bit platforms can still be correct.
      
      For 64bit platform, this problem still exists. The major reason is bit pos 8 times of byte pos. When proto-max-bulk-len is very larger, bit pos may overflow.
      But at 64bit platform, we don't have so long string. So this bug may never happen.
      
      Additionally this commit add a test cost `512MB` memory which is tag as `large-memory`. Make freebsd ci and valgrind ci ignore this test.
      71d45287
  3. 14 Jul, 2021 1 commit
    • Oran Agra's avatar
      Test infra, handle RESP3 attributes and big-numbers and bools (#9235) · 6a5bac30
      Oran Agra authored
      - promote the code in DEBUG PROTOCOL to addReplyBigNum
      - DEBUG PROTOCOL ATTRIB skips the attribute when client is RESP2
      - networking.c addReply for push and attributes generate assertion when
        called on a RESP2 client, anything else would produce a broken
        protocol that clients can't handle.
      6a5bac30
  4. 11 Jul, 2021 1 commit
    • perryitay's avatar
      Fail EXEC command in case a watched key is expired (#9194) · ac8b1df8
      perryitay authored
      
      
      There are two issues fixed in this commit: 
      1. we want to fail the EXEC command in case there is a watched key that's logically
         expired but not yet deleted by active expire or lazy expire.
      2. we saw that currently cache time is update in every `call()` (including nested calls),
         this time is being also being use for the isKeyExpired comparison, we want to update
         the cache time only in the first call (execCommand)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ac8b1df8
  5. 03 Jul, 2021 1 commit
  6. 01 Jul, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix CLIENT UNBLOCK crashing modules. (#9167) · aa139e2f
      Yossi Gottlieb authored
      Modules that use background threads with thread safe contexts are likely
      to use RM_BlockClient() without a timeout function, because they do not
      set up a timeout.
      
      Before this commit, `CLIENT UNBLOCK` would result with a crash as the
      `NULL` timeout callback is called. Beyond just crashing, this is also
      logically wrong as it may throw the module into an unexpected client
      state.
      
      This commits makes `CLIENT UNBLOCK` on such clients behave the same as
      any other client that is not in a blocked state and therefore cannot be
      unblocked.
      aa139e2f
  7. 24 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add bind-source-addr configuration argument. (#9142) · f233c4c5
      Yossi Gottlieb authored
      In the past, the first bind address that was explicitly specified was
      also used to bind outgoing connections. This could result with some
      problems. For example: on some systems using `bind 127.0.0.1` would
      result with outgoing connections also binding to `127.0.0.1` and failing
      to connect to remote addresses.
      
      With the recent change to the way `bind` is handled, this presented
      other issues:
      
      * The default first bind address is '*' which is not a valid address.
      * We make no distinction between user-supplied config that is identical
      to the default, and the default config.
      
      This commit addresses both these issues by introducing an explicit
      configuration parameter to control the bind address on outgoing
      connections.
      f233c4c5
  8. 22 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve bind and protected-mode config handling. (#9034) · 07b0d144
      Yossi Gottlieb authored
      * Specifying an empty `bind ""` configuration prevents Redis from listening on any TCP port. Before this commit, such configuration was not accepted.
      * Using `CONFIG GET bind` will always return an explicit configuration value. Before this commit, if a bind address was not specified the returned value was empty (which was an anomaly).
      
      Another behavior change is that modifying the `bind` configuration to a non-default value will NO LONGER DISABLE protected-mode implicitly.
      07b0d144
  9. 16 Jun, 2021 3 commits
    • yoav-steinberg's avatar
      Remove gopher protocol support. (#9057) · 362786c5
      yoav-steinberg authored
      Gopher support was added mainly because it was simple (trivial to add).
      But apparently even something that was trivial at the time, does cause complications
      down the line when adding more features.
      We recently ran into a few issues with io-threads conflicting with the gopher support.
      We had to either complicate the code further in order to solve them, or drop gopher.
      AFAIK it's completely unused, so we wanna chuck it, rather than keep supporting it.
      362786c5
    • chenyang8094's avatar
      Enhance mem_usage/free_effort/unlink/copy callbacks and add GetDbFromIO api. (#8999) · e0cd3ad0
      chenyang8094 authored
      Create new module type enhanced callbacks: mem_usage2, free_effort2, unlink2, copy2.
      These will be given a context point from which the module can obtain the key name and database id.
      In addition the digest and defrag context can now be used to obtain the key name and database id.
      e0cd3ad0
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  10. 15 Jun, 2021 1 commit
    • sundb's avatar
      Fix the wrong reisze of querybuf (#9003) · e5d8a5eb
      sundb authored
      The initialize memory of `querybuf` is `PROTO_IOBUF_LEN(1024*16) * 2` (due to sdsMakeRoomFor being greedy), under `jemalloc`, the allocated memory will be 40k.
      This will most likely result in the `querybuf` being resized when call `clientsCronResizeQueryBuffer` unless the client requests it fast enough.
      
      Note that this bug existed even before #7875, since the condition for resizing includes the sds headers (32k+6).
      
      ## Changes
      1. Use non-greedy sdsMakeRoomFor when allocating the initial query buffer (of 16k).
      1. Also use non-greedy allocation when working with BIG_ARG (we won't use that extra space anyway)
      2. in case we did use a greedy allocation, read as much as we can into the buffer we got (including internal frag), to reduce system calls.
      3. introduce a dedicated constant for the shrinking (same value as before)
      3. Add test for querybuf.
      4. improve a maxmemory test by ignoring the effect of replica query buffers (can accumulate many ACKs on slow env)
      5. improve a maxmemory by disabling slowlog (it will cause slight memory growth on slow env).
      e5d8a5eb
  11. 14 Jun, 2021 1 commit
    • YaacovHazan's avatar
      cleanup around loadAppendOnlyFile (#9012) · 1677efb9
      YaacovHazan authored
      Today when we load the AOF on startup, the loadAppendOnlyFile checks if
      the file is openning for reading.
      This check is redundent (dead code) as we open the AOF file for writing at initServer,
      and the file will always be existing for the loadAppendOnlyFile.
      
      In this commit:
      - remove all the exit(1) from loadAppendOnlyFile, as it is the caller
        responsibility to decide what to do in case of failure.
      - move the opening of the AOF file for writing, to be after we loading it.
      - avoid return -ERR in DEBUG LOADAOF, when the AOF is existing but empty
      1677efb9
  12. 10 Jun, 2021 2 commits
    • ZhaolongLi's avatar
      63da66bb
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  13. 08 Jun, 2021 1 commit
  14. 02 Jun, 2021 1 commit
  15. 01 Jun, 2021 1 commit
  16. 30 May, 2021 2 commits
    • Wang Yuan's avatar
      Make full use of aofrwblock's buf (#8975) · 58a03eca
      Wang Yuan authored
      Make aof rewrite buffer memory size more accurate, before, there may be 20%
      deviation with its real memory usage.
      
      The implication are both lower memory usage, and also a more accurate INFO.
      58a03eca
    • ny0312's avatar
      Always replicate time-to-live(TTL) as absolute timestamps in milliseconds (#8474) · 53d1acd5
      ny0312 authored
      Till now, on replica full-sync we used to transfer absolute time for TTL,
      however when a command arrived (EXPIRE or EXPIREAT),
      we used to propagate it as is to replicas (possibly with relative time),
      but always translate it to EXPIREAT (absolute time) to AOF.
      
      This commit changes that and will always use absolute time for propagation.
      see discussion in #8433
      
      Furthermore, we Introduce new commands: `EXPIRETIME/PEXPIRETIME`
      that allow extracting the absolute TTL time from a key.
      53d1acd5
  17. 20 May, 2021 1 commit
  18. 19 May, 2021 1 commit
  19. 17 May, 2021 1 commit
  20. 13 May, 2021 1 commit
  21. 04 May, 2021 1 commit
  22. 29 Apr, 2021 1 commit
  23. 22 Apr, 2021 1 commit
  24. 20 Apr, 2021 1 commit
  25. 19 Apr, 2021 1 commit
    • Hanna Fadida's avatar
      Modules: adding a module type for key space notification (#8759) · 53a4d6c3
      Hanna Fadida authored
      Adding a new type mask ​for key space notification, REDISMODULE_NOTIFY_MODULE, to enable unique notifications from commands on REDISMODULE_KEYTYPE_MODULE type keys (which is currently unsupported).
      
      Modules can subscribe to a module key keyspace notification by RM_SubscribeToKeyspaceEvents,
      and clients by notify-keyspace-events of redis.conf or via the CONFIG SET, with the characters 'd' or 'A' 
      (REDISMODULE_NOTIFY_MODULE type mask is part of the '**A**ll' notation for key space notifications).
      
      Refactor: move some pubsub test infra from pubsub.tcl to util.tcl to be re-used by other tests.
      53a4d6c3
  26. 15 Apr, 2021 1 commit
    • guybe7's avatar
      Add a timeout mechanism for replicas stuck in fullsync (#8762) · d63d0260
      guybe7 authored
      Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork
      child so that the parent is the one sending data to the replicas.
      This mechanism has an issue in which a hung replica will cause the master to wait
      for it to read the data sent to it forever, thus preventing the fork child from terminating
      and preventing the creations of any other forks.
      
      This PR adds a timeout mechanism, much like the ACK-based timeout,
      we disconnect replicas that aren't reading the RDB file fast enough.
      d63d0260
  27. 11 Apr, 2021 1 commit
    • Wang Yuan's avatar
      Fix wrong check for aof fsync and handle aof fsync errno (#8751) · a0e19e3c
      Wang Yuan authored
      The bio aof fsync fd may be closed by main thread (AOFRW done handler)
      and even possibly reused for another socket, pipe, or file.
      This can can an EBADF or EINVAL fsync error, which will lead to -MISCONF errors failing all writes.
      We just ignore these errno because aof fsync did not really fail.
      
      We handle errno when fsyncing aof in bio, so we could know the real reason
      when users get -MISCONF Errors writing to the AOF file error
      
      Issue created with #8419
      a0e19e3c
  28. 05 Apr, 2021 1 commit
    • Huang Zhw's avatar
      Fix "default" and overwritten / reset users will not have pubsub channels... · 3b74b550
      Huang Zhw authored
      Fix "default" and overwritten / reset users will not have pubsub channels permissions by default. (#8723)
      
      Background:
      Redis 6.2 added ACL control for pubsub channels (#7993), which were supposed
      to be permissive by default to retain compatibility with redis 6.0 ACL. 
      But due to a bug, only newly created users got this `acl-pubsub-default` applied,
      while overwritten (updated) users got reset to `resetchannels` (denied).
      
      Since the "default" user exists before loading the config file,
      any ACL change to it, results in an update / overwrite.
      
      So when a "default" user is loaded from config file or include ACL
      file with no channels related rules, the user will not have any
      permissions to any channels. But other users will have default
      permissions to any channels.
      
      When upgraded from 6.0 with config rewrite, this will lead to
      "default" user channels permissions lost.
      When users are loaded from include file, then call "acl load", users
      will also lost channels permissions.
      
      Similarly, the `reset` ACL rule, would have reset the user to be denied
      access to any channels, ignoring `acl-pubsub-default` and breaking
      compatibility with redis 6.0.
      
      The implication of this fix is that it regains compatibility with redis 6.0,
      but breaks compatibility with redis 6.2.0 and 2.0.1. e.g. after the upgrade,
      the default user will regain access to pubsub channels.
      
      Other changes:
      Additionally this commit rename server.acl_pubusub_default to
      server.acl_pubsub_default and fix typo in acl tests.
      3b74b550
  29. 04 Apr, 2021 1 commit
    • Sokolov Yura's avatar
      Add cluster-allow-replica-migration option. (#5285) · 1cab9620
      Sokolov Yura authored
      
      
      Previously (and by default after commit) when master loose its last slot
      (due to migration, for example), its replicas will migrate to new last slot
      holder.
      
      There are cases where this is not desired:
      * Consolidation that results with removed nodes (including the replica, eventually).
      * Manually configured cluster topologies, which the admin wishes to preserve.
      
      Needlessly migrating a replica triggers a full synchronization and can have a negative impact, so
      we prefer to be able to avoid it where possible.
      
      This commit adds 'cluster-allow-replica-migration' configuration option that is
      enabled by default to preserve existed behavior. When disabled, replicas will
      not be auto-migrated.
      
      Fixes #4896
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1cab9620
  30. 01 Apr, 2021 2 commits
    • Wang Yuan's avatar
      Handle remaining fsync errors (#8419) · 1eb85249
      Wang Yuan authored
      In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail.
      In `cluster.c`, we now return error, the calling function already handles these write errors.
      In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed.
      In `rio.c`, we now treat fsync errors like we do for write errors. 
      In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail.
      In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error,  moreover also set INFO command field `aof_last_write_status` to error.
      1eb85249
    • Wen Hui's avatar
      generalize config file check for sentinel (#8730) · d5935bb0
      Wen Hui authored
      The implications of this change is just that in the past when a config file was missing,
      in some cases it was exiting before printing the sever startup prints and sometimes after,
      and now it'll always exit before printing them.
      d5935bb0
  31. 30 Mar, 2021 2 commits
    • Jérôme Loyet's avatar
      Add replica-announced config option (#8653) · 91f4f416
      Jérôme Loyet authored
      The 'sentinel replicas <master>' command will ignore replicas with
      `replica-announced` set to no.
      
      The goal of disabling the config setting replica-announced is to allow ghost
      replicas. The replica is in the cluster, synchronize with its master, can be
      promoted to master and is not exposed to sentinel clients. This way, it is
      acting as a live backup or living ghost.
      
      In addition, to prevent the replica to be promoted as master, set
      replica-priority to 0.
      91f4f416
    • Viktor Söderqvist's avatar
      Add support for plaintext clients in TLS cluster (#8587) · 5629dbe7
      Viktor Söderqvist authored
      The cluster bus is established over TLS or non-TLS depending on the configuration tls-cluster. The client ports distributed in the cluster and sent to clients are assumed to be TLS or non-TLS also depending on tls-cluster.
      
      The cluster bus is now extended to also contain the non-TLS port of clients in a TLS cluster, when available. The non-TLS port of a cluster node, when available, is sent to clients connected without TLS in responses to CLUSTER SLOTS, CLUSTER NODES, CLUSTER SLAVES and MOVED and ASK redirects, instead of the TLS port.
      
      The user was able to override the client port by defining cluster-announce-port. Now cluster-announce-tls-port is added, so the user can define an alternative announce port for both TLS and non-TLS clients.
      
      Fixes #8134
      5629dbe7
  32. 26 Mar, 2021 1 commit
    • Huang Zhw's avatar
      make processCommand check publish channel permissions. (#8534) · e138698e
      Huang Zhw authored
      Add publish channel permissions check in processCommand.
      
      processCommand didn't check publish channel permissions, so we can
      queue a publish command in a transaction. But when exec the transaction,
      it will fail with -NOPERM.
      
      We also union keys/commands/channels permissions check togegher in
      ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
      publishCommand/subscribeCommand/psubscribeCommand. Always 
      check permissions in processCommand/execCommand/
      luaRedisGenericCommand.
      e138698e
  33. 25 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix SLOWLOG for blocked commands (#8632) · 497351ad
      Oran Agra authored
      * SLOWLOG didn't record anything for blocked commands because the client
        was reset and argv was already empty. there was a fix for this issue
        specifically for modules, now it works for all blocked clients.
      * The original command argv (before being re-written) was also reset
        before adding the slowlog on behalf of the blocked command.
      * Latency monitor is now updated regardless of the slowlog flags of the
        command or its execution (their purpose is to hide sensitive info from
        the slowlog, not hide the fact the latency happened).
      * Latency monitor now uses real_cmd rather than c->cmd (which may be
        different if the command got re-written, e.g. GEOADD)
      
      Changes:
      * Unify shared code between slowlog insertion in call() and
        updateStatsOnUnblock(), hopefully prevent future bugs from happening
        due to the later being overlooked.
      * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
        processing.
      * Add a test for SLOWLOG and BLPOP
      
      Notes:
      - real_cmd == c->lastcmd, except inside MULTI and Lua.
      - blocked commands never happen in these cases (MULTI / Lua)
      - real_cmd == c->cmd, except for when the command is rewritten (e.g.
        GEOADD)
      - blocked commands (currently) are never rewritten
      - other than the command's CLIENT_PREVENT_LOGGING, and the
        execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
        avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
        be off when executed from execCommand that runs from an AOF)
      497351ad
  34. 24 Mar, 2021 1 commit