1. 21 Jul, 2021 2 commits
    • Oran Agra's avatar
      Test infra, handle RESP3 attributes and big-numbers and bools (#9235) · 6cd84b64
      Oran Agra authored
      - promote the code in DEBUG PROTOCOL to addReplyBigNum
      - DEBUG PROTOCOL ATTRIB skips the attribute when client is RESP2
      - networking.c addReply for push and attributes generate assertion when
        called on a RESP2 client, anything else would produce a broken
        protocol that clients can't handle.
      
      (cherry picked from commit 6a5bac30)
      6cd84b64
    • perryitay's avatar
      Fail EXEC command in case a watched key is expired (#9194) · 3f4f9b63
      perryitay authored
      
      
      There are two issues fixed in this commit: 
      1. we want to fail the EXEC command in case there is a watched key that's logically
         expired but not yet deleted by active expire or lazy expire.
      2. we saw that currently cache time is update in every `call()` (including nested calls),
         this time is being also being use for the isKeyExpired comparison, we want to update
         the cache time only in the first call (execCommand)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      (cherry picked from commit ac8b1df8)
      3f4f9b63
  2. 01 Jun, 2021 2 commits
  3. 03 May, 2021 2 commits
  4. 19 Apr, 2021 1 commit
    • Hanna Fadida's avatar
      Modules: adding a module type for key space notification (#8759) · 53a4d6c3
      Hanna Fadida authored
      Adding a new type mask ​for key space notification, REDISMODULE_NOTIFY_MODULE, to enable unique notifications from commands on REDISMODULE_KEYTYPE_MODULE type keys (which is currently unsupported).
      
      Modules can subscribe to a module key keyspace notification by RM_SubscribeToKeyspaceEvents,
      and clients by notify-keyspace-events of redis.conf or via the CONFIG SET, with the characters 'd' or 'A' 
      (REDISMODULE_NOTIFY_MODULE type mask is part of the '**A**ll' notation for key space notifications).
      
      Refactor: move some pubsub test infra from pubsub.tcl to util.tcl to be re-used by other tests.
      53a4d6c3
  5. 15 Apr, 2021 1 commit
    • guybe7's avatar
      Add a timeout mechanism for replicas stuck in fullsync (#8762) · d63d0260
      guybe7 authored
      Starting redis 6.0 (part of the TLS feature), diskless master uses pipe from the fork
      child so that the parent is the one sending data to the replicas.
      This mechanism has an issue in which a hung replica will cause the master to wait
      for it to read the data sent to it forever, thus preventing the fork child from terminating
      and preventing the creations of any other forks.
      
      This PR adds a timeout mechanism, much like the ACK-based timeout,
      we disconnect replicas that aren't reading the RDB file fast enough.
      d63d0260
  6. 11 Apr, 2021 1 commit
    • Wang Yuan's avatar
      Fix wrong check for aof fsync and handle aof fsync errno (#8751) · a0e19e3c
      Wang Yuan authored
      The bio aof fsync fd may be closed by main thread (AOFRW done handler)
      and even possibly reused for another socket, pipe, or file.
      This can can an EBADF or EINVAL fsync error, which will lead to -MISCONF errors failing all writes.
      We just ignore these errno because aof fsync did not really fail.
      
      We handle errno when fsyncing aof in bio, so we could know the real reason
      when users get -MISCONF Errors writing to the AOF file error
      
      Issue created with #8419
      a0e19e3c
  7. 05 Apr, 2021 1 commit
    • Huang Zhw's avatar
      Fix "default" and overwritten / reset users will not have pubsub channels... · 3b74b550
      Huang Zhw authored
      Fix "default" and overwritten / reset users will not have pubsub channels permissions by default. (#8723)
      
      Background:
      Redis 6.2 added ACL control for pubsub channels (#7993), which were supposed
      to be permissive by default to retain compatibility with redis 6.0 ACL. 
      But due to a bug, only newly created users got this `acl-pubsub-default` applied,
      while overwritten (updated) users got reset to `resetchannels` (denied).
      
      Since the "default" user exists before loading the config file,
      any ACL change to it, results in an update / overwrite.
      
      So when a "default" user is loaded from config file or include ACL
      file with no channels related rules, the user will not have any
      permissions to any channels. But other users will have default
      permissions to any channels.
      
      When upgraded from 6.0 with config rewrite, this will lead to
      "default" user channels permissions lost.
      When users are loaded from include file, then call "acl load", users
      will also lost channels permissions.
      
      Similarly, the `reset` ACL rule, would have reset the user to be denied
      access to any channels, ignoring `acl-pubsub-default` and breaking
      compatibility with redis 6.0.
      
      The implication of this fix is that it regains compatibility with redis 6.0,
      but breaks compatibility with redis 6.2.0 and 2.0.1. e.g. after the upgrade,
      the default user will regain access to pubsub channels.
      
      Other changes:
      Additionally this commit rename server.acl_pubusub_default to
      server.acl_pubsub_default and fix typo in acl tests.
      3b74b550
  8. 04 Apr, 2021 1 commit
    • Sokolov Yura's avatar
      Add cluster-allow-replica-migration option. (#5285) · 1cab9620
      Sokolov Yura authored
      
      
      Previously (and by default after commit) when master loose its last slot
      (due to migration, for example), its replicas will migrate to new last slot
      holder.
      
      There are cases where this is not desired:
      * Consolidation that results with removed nodes (including the replica, eventually).
      * Manually configured cluster topologies, which the admin wishes to preserve.
      
      Needlessly migrating a replica triggers a full synchronization and can have a negative impact, so
      we prefer to be able to avoid it where possible.
      
      This commit adds 'cluster-allow-replica-migration' configuration option that is
      enabled by default to preserve existed behavior. When disabled, replicas will
      not be auto-migrated.
      
      Fixes #4896
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1cab9620
  9. 01 Apr, 2021 2 commits
    • Wang Yuan's avatar
      Handle remaining fsync errors (#8419) · 1eb85249
      Wang Yuan authored
      In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail.
      In `cluster.c`, we now return error, the calling function already handles these write errors.
      In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed.
      In `rio.c`, we now treat fsync errors like we do for write errors. 
      In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail.
      In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error,  moreover also set INFO command field `aof_last_write_status` to error.
      1eb85249
    • Wen Hui's avatar
      generalize config file check for sentinel (#8730) · d5935bb0
      Wen Hui authored
      The implications of this change is just that in the past when a config file was missing,
      in some cases it was exiting before printing the sever startup prints and sometimes after,
      and now it'll always exit before printing them.
      d5935bb0
  10. 30 Mar, 2021 2 commits
    • Jérôme Loyet's avatar
      Add replica-announced config option (#8653) · 91f4f416
      Jérôme Loyet authored
      The 'sentinel replicas <master>' command will ignore replicas with
      `replica-announced` set to no.
      
      The goal of disabling the config setting replica-announced is to allow ghost
      replicas. The replica is in the cluster, synchronize with its master, can be
      promoted to master and is not exposed to sentinel clients. This way, it is
      acting as a live backup or living ghost.
      
      In addition, to prevent the replica to be promoted as master, set
      replica-priority to 0.
      91f4f416
    • Viktor Söderqvist's avatar
      Add support for plaintext clients in TLS cluster (#8587) · 5629dbe7
      Viktor Söderqvist authored
      The cluster bus is established over TLS or non-TLS depending on the configuration tls-cluster. The client ports distributed in the cluster and sent to clients are assumed to be TLS or non-TLS also depending on tls-cluster.
      
      The cluster bus is now extended to also contain the non-TLS port of clients in a TLS cluster, when available. The non-TLS port of a cluster node, when available, is sent to clients connected without TLS in responses to CLUSTER SLOTS, CLUSTER NODES, CLUSTER SLAVES and MOVED and ASK redirects, instead of the TLS port.
      
      The user was able to override the client port by defining cluster-announce-port. Now cluster-announce-tls-port is added, so the user can define an alternative announce port for both TLS and non-TLS clients.
      
      Fixes #8134
      5629dbe7
  11. 26 Mar, 2021 1 commit
    • Huang Zhw's avatar
      make processCommand check publish channel permissions. (#8534) · e138698e
      Huang Zhw authored
      Add publish channel permissions check in processCommand.
      
      processCommand didn't check publish channel permissions, so we can
      queue a publish command in a transaction. But when exec the transaction,
      it will fail with -NOPERM.
      
      We also union keys/commands/channels permissions check togegher in
      ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
      publishCommand/subscribeCommand/psubscribeCommand. Always 
      check permissions in processCommand/execCommand/
      luaRedisGenericCommand.
      e138698e
  12. 25 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix SLOWLOG for blocked commands (#8632) · 497351ad
      Oran Agra authored
      * SLOWLOG didn't record anything for blocked commands because the client
        was reset and argv was already empty. there was a fix for this issue
        specifically for modules, now it works for all blocked clients.
      * The original command argv (before being re-written) was also reset
        before adding the slowlog on behalf of the blocked command.
      * Latency monitor is now updated regardless of the slowlog flags of the
        command or its execution (their purpose is to hide sensitive info from
        the slowlog, not hide the fact the latency happened).
      * Latency monitor now uses real_cmd rather than c->cmd (which may be
        different if the command got re-written, e.g. GEOADD)
      
      Changes:
      * Unify shared code between slowlog insertion in call() and
        updateStatsOnUnblock(), hopefully prevent future bugs from happening
        due to the later being overlooked.
      * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
        processing.
      * Add a test for SLOWLOG and BLPOP
      
      Notes:
      - real_cmd == c->lastcmd, except inside MULTI and Lua.
      - blocked commands never happen in these cases (MULTI / Lua)
      - real_cmd == c->cmd, except for when the command is rewritten (e.g.
        GEOADD)
      - blocked commands (currently) are never rewritten
      - other than the command's CLIENT_PREVENT_LOGGING, and the
        execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
        avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
        be off when executed from execCommand that runs from an AOF)
      497351ad
  13. 24 Mar, 2021 1 commit
  14. 22 Mar, 2021 2 commits
  15. 16 Mar, 2021 1 commit
  16. 14 Mar, 2021 1 commit
  17. 10 Mar, 2021 2 commits
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
    • guybe7's avatar
      Cleanup ZADD_* flags (#8559) · 61a73de6
      guybe7 authored
      Have a clear separation between in and out flags
      
      Other changes:
      
      delete dead code in RM_ZsetIncrby: if zsetAdd returned error (happens only if
      the result of the operation is NAN or if score is NAN) we return immediately so
      there is no way that zsetAdd succeeded and returned NAN in the out-flags
      61a73de6
  18. 05 Mar, 2021 1 commit
  19. 03 Mar, 2021 1 commit
  20. 02 Mar, 2021 1 commit
  21. 01 Mar, 2021 1 commit
    • YaacovHazan's avatar
      Make port, tls-port and bind configurations modifiable (#8510) · a031d268
      YaacovHazan authored
      Add ability to modify port, tls-port and bind configurations by CONFIG SET command.
      
      To simplify the code and make it cleaner, a new structure
      added, socketFds, which contains the file descriptors array and its counter,
      and used for TCP, TLS and Cluster sockets file descriptors.
      a031d268
  22. 28 Feb, 2021 1 commit
    • Viktor Söderqvist's avatar
      Shared reusable client for RM_Call() (#8516) · 6122f1c4
      Viktor Söderqvist authored
      A single client pointer is added in the server struct. This is
      initialized by the first RM_Call() and reused for every subsequent
      RM_Call() except if it's already in use, which means that it's not
      used for (recursive) module calls to modules. For these, a new
      "fake" client is created each time.
      
      Other changes:
      * Avoid allocating a dict iterator in pubsubUnsubscribeAllChannels
        when not needed
      6122f1c4
  23. 26 Feb, 2021 1 commit
  24. 21 Feb, 2021 1 commit
  25. 17 Feb, 2021 1 commit
  26. 16 Feb, 2021 1 commit
    • uriyage's avatar
      Adds INFO fields to track fork child progress (#8414) · fd052d2a
      uriyage authored
      * Adding current_save_keys_total and current_save_keys_processed info fields.
        Present in replication, BGSAVE and AOFRW.
      * Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
      * Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
        and module forks.
      fd052d2a
  27. 15 Feb, 2021 1 commit
  28. 09 Feb, 2021 1 commit
  29. 29 Jan, 2021 2 commits
    • filipe oliveira's avatar
      Enabled background and reply time tracking on blocked on keys/blocked on... · f0c5052a
      filipe oliveira authored
      Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
      
      This commit enables tracking time of the background tasks and on replies,
      opening the door for properly tracking commands that rely on blocking / background
       work via the slowlog, latency history, and commandstats. 
      
      Some notes:
      - The time spent blocked waiting for key changes, or blocked on synchronous
        replication is not accounted for. 
      
      - **This commit does not affect latency tracking of commands that are non-blocking
        or do not have background work.** ( meaning that it all stays the same with exception to
        `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
        on background threads ). 
      
      -  Specifically for latency history command we've added a new event class named
        `command-unblocking` that will enable latency monitoring on commands that spawn
        background threads to do the work.
      
      - For blocking commands we're now considering the total time of a command as the
        time spent on call() + the time spent on replying when unblocked.
      
      - For Modules commands that rely on background threads we're now considering the
        total time of a command as the time spent on call (main thread) + the time spent on
        the background thread ( if marked within `RedisModule_MeasureTimeStart()` and
        `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread)
      
      To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on
      a module that blocks the client and sleeps on the background for a given time. 
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout
      - check blocked command with multiple calls RedisModule_MeasureTimeStart()  is tracking the total background time
      - check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
      f0c5052a
    • Yang Bodong's avatar
      Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297) · b9a0500f
      Yang Bodong authored
      
      
      New commands:
      `HRANDFIELD [<count> [WITHVALUES]]`
      `ZRANDMEMBER [<count> [WITHSCORES]]`
      Algorithms are similar to the one in SRANDMEMBER.
      
      Both return a simple bulk response when no arguments are given, and an array otherwise.
      In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
      note: in all 3 commands, the only option that also provides random order is the one with negative count.
      
      Changes to SRANDMEMBER
      * Optimization when count is 1, we can use the more efficient algorithm of non-unique random
      * optimization: work with sds strings rather than robj
      
      Other changes:
      * zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
        case the buffer is too small)
      * Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
        case 4) and by accident used a negative count
      Co-authored-by: default avatarxinluton <xinluton@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b9a0500f
  30. 28 Jan, 2021 3 commits
    • Allen Farris's avatar
      implement FAILOVER command (#8315) · 0d18a1e8
      Allen Farris authored
      Implement FAILOVER command, which coordinates failover
      between the server and one of its replicas.
      0d18a1e8
    • Yossi Gottlieb's avatar
      Add proc-title-template option. (#8397) · 4bb5ccbe
      Yossi Gottlieb authored
      Make it possible to customize the process title, i.e. include custom
      strings, immutable configuration like port, tls-port, unix socket name,
      etc.
      4bb5ccbe
    • Z. Liu's avatar
      Add 'set-proc-title' config so that this mechanism can be disabled (#3623) · 17b34c73
      Z. Liu authored
      
      
      if option `set-proc-title' is no, then do nothing for proc title.
      
      The reason has been explained long ago, see following:
      
      We update redis to 2.8.8, then found there are some side effect when
      redis always change the process title.
      
      We run several slave instance on one computer, and all these salves
      listen on unix socket only, then ps will show:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0
      
      for redis 2.6 the output of ps is like following:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf
      
      Later is more informational in our case. The situation
      is worse when we manage the config and process running
      state by salt. Salt check the process by running "ps |
      grep SIG" (for Gentoo System) to check the running
      state, where SIG is the string to search for when
      looking for the service process with ps. Previously, we
      define sig as "/usr/sbin/redis-server
      /etc/redis/a.conf". Since the ps output is identical for
      our case, so we have no way to check the state of
      specified redis instance.
      
      So, for our case, we prefer the old behavior, i.e, do
      not change the process title for the main redis process.
      Or add an option such as "set-proc-title [yes|no]" to
      control this behavior.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      17b34c73