1. 11 Jul, 2021 1 commit
    • perryitay's avatar
      Fail EXEC command in case a watched key is expired (#9194) · ac8b1df8
      perryitay authored
      
      
      There are two issues fixed in this commit: 
      1. we want to fail the EXEC command in case there is a watched key that's logically
         expired but not yet deleted by active expire or lazy expire.
      2. we saw that currently cache time is update in every `call()` (including nested calls),
         this time is being also being use for the isKeyExpired comparison, we want to update
         the cache time only in the first call (execCommand)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ac8b1df8
  2. 09 Jul, 2021 1 commit
  3. 05 Jul, 2021 2 commits
    • Oran Agra's avatar
      Query buffer shrinking improvements (#5013) · ec582cc7
      Oran Agra authored
      
      
      when tracking the peak, don't reset the peak to 0, reset it to the
      maximum of the current used, and the planned to be used by the current
      arg.
      
      when shrining, split the two separate conditions.
      the idle time shrinking will remove all free space.
      but the peak based shrinking will keep room for the current arg.
      
      when we resize due to a peak (rahter than idle time), don't trim all
      unused space, let the qbuf keep a size that's sufficient for the
      currently process bulklen, and the current peak.
      Co-authored-by: default avatarsundb <sundbcn@gmail.com>
      Co-authored-by: default avataryoav-steinberg <yoav@monfort.co.il>
      ec582cc7
    • zhaozhao.zz's avatar
      resize query buffer more accurately · 2248eaac
      zhaozhao.zz authored
      1. querybuf_peak has not been updated correctly in readQueryFromClient.
      2. qbuf shrinking uses sdsalloc instead of sdsAllocSize
      
      see more details in issue #4983
      2248eaac
  4. 03 Jul, 2021 1 commit
  5. 01 Jul, 2021 1 commit
    • Wang Yuan's avatar
      Don't start in sentinel mode if only the folder name contains redis-sentinel (#9176) · 16e04ed9
      Wang Yuan authored
      Before this commit, redis-server starts in sentinel mode if the first startup
      argument has the string redis-sentinel, so redis also starts in sentinel mode
      if the directory it was started from contains the string redis-sentinel.
      Now we check the executable name instead of directory.
      
      Some examples:
      1. Execute ./redis-sentinel/redis/src/redis-sentinel, starts in sentinel mode.
      2. Execute ./redis-sentinel/redis/src/redis-server, starts in server mode,
         but before, redis will start in sentinel mode.
      3. Execute ./redis-sentinel/redis/src/redis-server --sentinel, of course, like
         before, starts in sentinel mode.
      16e04ed9
  6. 24 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Add bind-source-addr configuration argument. (#9142) · f233c4c5
      Yossi Gottlieb authored
      In the past, the first bind address that was explicitly specified was
      also used to bind outgoing connections. This could result with some
      problems. For example: on some systems using `bind 127.0.0.1` would
      result with outgoing connections also binding to `127.0.0.1` and failing
      to connect to remote addresses.
      
      With the recent change to the way `bind` is handled, this presented
      other issues:
      
      * The default first bind address is '*' which is not a valid address.
      * We make no distinction between user-supplied config that is identical
      to the default, and the default config.
      
      This commit addresses both these issues by introducing an explicit
      configuration parameter to control the bind address on outgoing
      connections.
      f233c4c5
  7. 22 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve bind and protected-mode config handling. (#9034) · 07b0d144
      Yossi Gottlieb authored
      * Specifying an empty `bind ""` configuration prevents Redis from listening on any TCP port. Before this commit, such configuration was not accepted.
      * Using `CONFIG GET bind` will always return an explicit configuration value. Before this commit, if a bind address was not specified the returned value was empty (which was an anomaly).
      
      Another behavior change is that modifying the `bind` configuration to a non-default value will NO LONGER DISABLE protected-mode implicitly.
      07b0d144
  8. 17 Jun, 2021 1 commit
  9. 16 Jun, 2021 1 commit
    • Uri Shachar's avatar
      Cleaning up the cluster interface by moving almost all related declar… (#9080) · c7e502a0
      Uri Shachar authored
      * Cleaning up the cluster interface by moving almost all related declarations into cluster.h
      (no logic change -- just moving declarations/definitions around)
      
      This initial effort leaves two items out of scope - the configuration parsing into the server
      struct and the internals exposed by the clusterNode struct.
      
      * Remove unneeded declarations of dictSds*
      Ideally all the dictSds functionality would move from server.c into a dedicated module
      so we can avoid the duplication in redis-benchmark/cli
      
      * Move crc16 back into server.h, will be moved out once we create a seperate header file for
      hashing functions
      c7e502a0
  10. 15 Jun, 2021 1 commit
    • sundb's avatar
      Fix the wrong reisze of querybuf (#9003) · e5d8a5eb
      sundb authored
      The initialize memory of `querybuf` is `PROTO_IOBUF_LEN(1024*16) * 2` (due to sdsMakeRoomFor being greedy), under `jemalloc`, the allocated memory will be 40k.
      This will most likely result in the `querybuf` being resized when call `clientsCronResizeQueryBuffer` unless the client requests it fast enough.
      
      Note that this bug existed even before #7875, since the condition for resizing includes the sds headers (32k+6).
      
      ## Changes
      1. Use non-greedy sdsMakeRoomFor when allocating the initial query buffer (of 16k).
      1. Also use non-greedy allocation when working with BIG_ARG (we won't use that extra space anyway)
      2. in case we did use a greedy allocation, read as much as we can into the buffer we got (including internal frag), to reduce system calls.
      3. introduce a dedicated constant for the shrinking (same value as before)
      3. Add test for querybuf.
      4. improve a maxmemory test by ignoring the effect of replica query buffers (can accumulate many ACKs on slow env)
      5. improve a maxmemory by disabling slowlog (it will cause slight memory growth on slow env).
      e5d8a5eb
  11. 14 Jun, 2021 1 commit
    • YaacovHazan's avatar
      cleanup around loadAppendOnlyFile (#9012) · 1677efb9
      YaacovHazan authored
      Today when we load the AOF on startup, the loadAppendOnlyFile checks if
      the file is openning for reading.
      This check is redundent (dead code) as we open the AOF file for writing at initServer,
      and the file will always be existing for the loadAppendOnlyFile.
      
      In this commit:
      - remove all the exit(1) from loadAppendOnlyFile, as it is the caller
        responsibility to decide what to do in case of failure.
      - move the opening of the AOF file for writing, to be after we loading it.
      - avoid return -ERR in DEBUG LOADAOF, when the AOF is existing but empty
      1677efb9
  12. 10 Jun, 2021 1 commit
    • Binbin's avatar
      Fixed some typos, add a spell check ci and others minor fix (#8890) · 0bfccc55
      Binbin authored
      This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes.
      This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything,
      but at least it is also unlikely to cause false positives.
      
      Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use.
      
      Here's a summary of other changes:
      1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces).
      2. Outdated function / variable / argument names in comments
      3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString.
      4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751
      5. Some outdated https link URLs.
      6. Fix some outdated comment. Such as:
          - In README: about the rdb, we used to said create a `thread`, change to `process`
          - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey)
          - notifyKeyspaceEvent fucntion comment (add type arg)
          - Some others minor fix in comment (Most of them are incorrectly quoted by variable names)
      7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
      0bfccc55
  13. 30 May, 2021 1 commit
    • ny0312's avatar
      Always replicate time-to-live(TTL) as absolute timestamps in milliseconds (#8474) · 53d1acd5
      ny0312 authored
      Till now, on replica full-sync we used to transfer absolute time for TTL,
      however when a command arrived (EXPIRE or EXPIREAT),
      we used to propagate it as is to replicas (possibly with relative time),
      but always translate it to EXPIREAT (absolute time) to AOF.
      
      This commit changes that and will always use absolute time for propagation.
      see discussion in #8433
      
      Furthermore, we Introduce new commands: `EXPIRETIME/PEXPIRETIME`
      that allow extracting the absolute TTL time from a key.
      53d1acd5
  14. 19 May, 2021 1 commit
  15. 17 May, 2021 1 commit
  16. 13 May, 2021 1 commit
  17. 04 May, 2021 1 commit
  18. 29 Apr, 2021 1 commit
  19. 27 Apr, 2021 1 commit
    • Oran Agra's avatar
      Prevent replicas from sending commands that interact with keyspace (#8868) · 46f4ebbe
      Oran Agra authored
      This solves an issue reported in #8712 in which a replica would bypass
      the client write pause check and cause an assertion due to executing a
      write command during failover.
      
      The fact is that we don't expect replicas to execute any command other
      than maybe REPLCONF and PING, etc. but matching against the ADMIN
      command flag is insufficient, so instead i just block keyspace access
      for now.
      46f4ebbe
  20. 21 Apr, 2021 1 commit
  21. 19 Apr, 2021 1 commit
    • Wen Hui's avatar
      fix invalid master_link_down_since_seconds in info repication (#8785) · 0413fbc7
      Wen Hui authored
      When replica never successfully connect to master, server.repl_down_since
      will be initialized to 0, therefore, the info master_link_down_since_seconds
      was showing the current unix timestamp, which does not make much sense.
      
      This commit fixes the issue by showing master_link_down_since_seconds to -1.
      means the replica never connect to master before.
      
      This commit also resets this variable back to 0 when a replica is turned into
      a master, so that it'll behave the same if the master is later turned into a
      replica again.
      
      The implication of this change is that if some app is checking if the value is > 60
      do something, like conclude the replica is stale, this could case harm (changing
      a big positive number with a small one).
      0413fbc7
  22. 13 Apr, 2021 1 commit
  23. 11 Apr, 2021 1 commit
    • Wang Yuan's avatar
      Fix wrong check for aof fsync and handle aof fsync errno (#8751) · a0e19e3c
      Wang Yuan authored
      The bio aof fsync fd may be closed by main thread (AOFRW done handler)
      and even possibly reused for another socket, pipe, or file.
      This can can an EBADF or EINVAL fsync error, which will lead to -MISCONF errors failing all writes.
      We just ignore these errno because aof fsync did not really fail.
      
      We handle errno when fsyncing aof in bio, so we could know the real reason
      when users get -MISCONF Errors writing to the AOF file error
      
      Issue created with #8419
      a0e19e3c
  24. 01 Apr, 2021 2 commits
    • Wang Yuan's avatar
      Handle remaining fsync errors (#8419) · 1eb85249
      Wang Yuan authored
      In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail.
      In `cluster.c`, we now return error, the calling function already handles these write errors.
      In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed.
      In `rio.c`, we now treat fsync errors like we do for write errors. 
      In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail.
      In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error,  moreover also set INFO command field `aof_last_write_status` to error.
      1eb85249
    • Wen Hui's avatar
      generalize config file check for sentinel (#8730) · d5935bb0
      Wen Hui authored
      The implications of this change is just that in the past when a config file was missing,
      in some cases it was exiting before printing the sever startup prints and sometimes after,
      and now it'll always exit before printing them.
      d5935bb0
  25. 30 Mar, 2021 1 commit
    • Jérôme Loyet's avatar
      Add replica-announced config option (#8653) · 91f4f416
      Jérôme Loyet authored
      The 'sentinel replicas <master>' command will ignore replicas with
      `replica-announced` set to no.
      
      The goal of disabling the config setting replica-announced is to allow ghost
      replicas. The replica is in the cluster, synchronize with its master, can be
      promoted to master and is not exposed to sentinel clients. This way, it is
      acting as a live backup or living ghost.
      
      In addition, to prevent the replica to be promoted as master, set
      replica-priority to 0.
      91f4f416
  26. 26 Mar, 2021 1 commit
    • Huang Zhw's avatar
      make processCommand check publish channel permissions. (#8534) · e138698e
      Huang Zhw authored
      Add publish channel permissions check in processCommand.
      
      processCommand didn't check publish channel permissions, so we can
      queue a publish command in a transaction. But when exec the transaction,
      it will fail with -NOPERM.
      
      We also union keys/commands/channels permissions check togegher in
      ACLCheckAllPerm. Remove pubsubCheckACLPermissionsOrReply in 
      publishCommand/subscribeCommand/psubscribeCommand. Always 
      check permissions in processCommand/execCommand/
      luaRedisGenericCommand.
      e138698e
  27. 25 Mar, 2021 1 commit
    • Oran Agra's avatar
      Fix SLOWLOG for blocked commands (#8632) · 497351ad
      Oran Agra authored
      * SLOWLOG didn't record anything for blocked commands because the client
        was reset and argv was already empty. there was a fix for this issue
        specifically for modules, now it works for all blocked clients.
      * The original command argv (before being re-written) was also reset
        before adding the slowlog on behalf of the blocked command.
      * Latency monitor is now updated regardless of the slowlog flags of the
        command or its execution (their purpose is to hide sensitive info from
        the slowlog, not hide the fact the latency happened).
      * Latency monitor now uses real_cmd rather than c->cmd (which may be
        different if the command got re-written, e.g. GEOADD)
      
      Changes:
      * Unify shared code between slowlog insertion in call() and
        updateStatsOnUnblock(), hopefully prevent future bugs from happening
        due to the later being overlooked.
      * Reset CLIENT_PREVENT_LOGGING in resetClient rather than after command
        processing.
      * Add a test for SLOWLOG and BLPOP
      
      Notes:
      - real_cmd == c->lastcmd, except inside MULTI and Lua.
      - blocked commands never happen in these cases (MULTI / Lua)
      - real_cmd == c->cmd, except for when the command is rewritten (e.g.
        GEOADD)
      - blocked commands (currently) are never rewritten
      - other than the command's CLIENT_PREVENT_LOGGING, and the
        execution flag CLIENT_PREVENT_LOGGING, other cases that we want to
        avoid slowlog are on AOF loading (specifically CMD_CALL_SLOWLOG will
        be off when executed from execCommand that runs from an AOF)
      497351ad
  28. 24 Mar, 2021 2 commits
  29. 22 Mar, 2021 1 commit
    • Yossi Gottlieb's avatar
      Fix slowdown due to child reporting CoW. (#8645) · c3df27d1
      Yossi Gottlieb authored
      Reading CoW from /proc/<pid>/smaps can be slow with large processes on
      some platforms.
      
      This measures the time it takes to read CoW info and limits the duty
      cycle of future updates to roughly 1/100.
      
      As current_cow_size no longer represnets a current, fixed interval value
      there is also a new current_cow_size_age field that provides information
      about the age of the size value, in seconds.
      c3df27d1
  30. 16 Mar, 2021 3 commits
    • Huang Zhw's avatar
      When tests exit normally, some processes may still be alive (#8647) · a19c4058
      Huang Zhw authored
      In certain scenario start_server may think it failed to start a redis server
      although it started successfully. in these cases, it'll not terminate it, and
      it'll remain running when the test is over.
      
      In start_server if config doesn't have bind (the minimal.conf in introspection.tcl),
      it will try to bind ipv4 and ipv6. One may success while other fails. It will
      output "Could not create server TCP listening socket".
      wait_server_started uses this message to check whether instance started
      successfully. So it will consider that it failed even though redis started successfully.
      
      Additionally, in some cases it wasn't clear to users why the server exited,
      since the warning message printed to the log, could in some cases be harmless,
      and in some cases fatal.
      
      This PR adds makes a clear distinction between a warning log message and
      a fatal one, and changes the test suite to look for the fatal message.
      a19c4058
    • Yossi Gottlieb's avatar
      Server won't start on alpine/libmusl without IPv6. (#8655) · df5f543b
      Yossi Gottlieb authored
      listenToPort attempts to gracefully handle and ignore certain errors but does not store errno prior to logging, which in turn calls several libc functions that may overwrite errno.
      
      This has been discovered due to libmusl strftime() always returning with errno set to EINVAL, which resulted with docker-library/redis#273.
      df5f543b
    • Madelyn Olson's avatar
      Redact slowlog entries for config with sensitive data. (#8584) · e1d98bca
      Madelyn Olson authored
      Redact config set requirepass/masterauth/masteruser from slowlog in addition to showing ACL commands without sensitive values. 
      e1d98bca
  31. 15 Mar, 2021 1 commit
    • guybe7's avatar
      Missing EXEC on modules propagation after failed EVAL execution (#8654) · dba33a94
      guybe7 authored
      1. moduleReplicateMultiIfNeeded should use server.in_eval like
         moduleHandlePropagationAfterCommandCallback
      2. server.in_eval could have been set to 1 and not reset back
         to 0 (a lot of missed early-exits after in_eval is already 1)
      
      Note: The new assertions in processCommand cover (2) and I added
      two module tests to cover (1)
      
      Implications:
      If an EVAL that failed (and thus left server.in_eval=1) runs before a module
      command that replicates, the replication stream will contain MULTI (because
      moduleReplicateMultiIfNeeded used to check server.lua_caller which is NULL
      at this point) but not EXEC (because server.in_eval==1)
      This only affects modules as module.c the only user of server.in_eval.
      
      Affects versions 6.2.0, 6.2.1
      dba33a94
  32. 14 Mar, 2021 1 commit
    • Guillem Jover's avatar
      Send the readiness notification when we are ready to accept connections (#8409) · 3a5905fa
      Guillem Jover authored
      On a replica we do accept connections, even though commands accessing
      the database will operate in read-only mode. But the server is still
      already operational and processing commands.
      
      Not sending the readiness notification means that on a HA setup where
      the nodes all start as replicas (with replicaof in the config) with
      a replica that cannot connect to the master server and which might not
      come back in a predictable amount of time or at all, the service
      supervisor will end up timing out the service and terminating it, with
      no option to promote it to be the main instance. This seems counter to
      what the readiness notification is supposed to be signaling.
      
      Instead send the readiness notification when we start accepting
      commands, and then send the various server status changes as that.
      
      Fixes: commit 641c64ad
      Fixes: commit dfb598cf
      3a5905fa
  33. 11 Mar, 2021 1 commit
  34. 10 Mar, 2021 2 commits
    • guybe7's avatar
      Fix some issues with modules and MULTI/EXEC (#8617) · 3d0b427c
      guybe7 authored
      Bug 1:
      When a module ctx is freed moduleHandlePropagationAfterCommandCallback
      is called and handles propagation. We want to prevent it from propagating
      commands that were not replicated by the same context. Example:
      1. module1.foo does: RM_Replicate(cmd1); RM_Call(cmd2); RM_Replicate(cmd3)
      2. RM_Replicate(cmd1) propagates MULTI and adds cmd1 to also_propagagte
      3. RM_Call(cmd2) create a new ctx, calls call() and destroys the ctx.
      4. moduleHandlePropagationAfterCommandCallback is called, calling
         alsoPropagates EXEC (Note: EXEC is still not written to socket),
         setting server.in_trnsaction = 0
      5. RM_Replicate(cmd3) is called, propagagting yet another MULTI (now
         we have nested MULTI calls, which is no good) and then cmd3
      
      We must prevent RM_Call(cmd2) from resetting server.in_transaction.
      REDISMODULE_CTX_MULTI_EMITTED was revived for that purpose.
      
      Bug 2:
      Fix issues with nested RM_Call where some have '!' and some don't.
      Example:
      1. module1.foo does RM_Call of module2.bar without replication (i.e. no '!')
      2. module2.bar internally calls RM_Call of INCR with '!'
      3. at the end of module1.foo we call RM_ReplicateVerbatim
      
      We want the replica/AOF to see only module1.foo and not the INCR from module2.bar
      
      Introduced a global replication_allowed flag inside RM_Call to determine
      whether we need to replicate or not (even if '!' was specified)
      
      Other changes:
      Split beforePropagateMultiOrExec to beforePropagateMulti afterPropagateExec
      just for better readability
      3d0b427c
    • Oran Agra's avatar
      strip % sign from current_fork_perc info field (#8628) · 7778f1b4
      Oran Agra authored
      `master_sync_perc` and `loading_loaded_perc` don't have that sign,
      and i think the info field should be a raw floating point number (the name suggests its units).
      
      we already have `used_memory_peak_perc` and `used_memory_dataset_perc` which do add the `%` sign, but:
      1) i think it was a mistake but maybe too late to fix now, and maybe not too late to fix for `current_fork_perc`
      2) it is more important to be consistent with the two other "progress" "prec" metrics, and not with the "utilization" metric.
      7778f1b4