1. 01 Mar, 2021 1 commit
    • YaacovHazan's avatar
      Make port, tls-port and bind configurations modifiable (#8510) · a031d268
      YaacovHazan authored
      Add ability to modify port, tls-port and bind configurations by CONFIG SET command.
      
      To simplify the code and make it cleaner, a new structure
      added, socketFds, which contains the file descriptors array and its counter,
      and used for TCP, TLS and Cluster sockets file descriptors.
      a031d268
  2. 26 Feb, 2021 1 commit
  3. 24 Feb, 2021 1 commit
  4. 21 Feb, 2021 1 commit
  5. 17 Feb, 2021 1 commit
  6. 16 Feb, 2021 1 commit
    • uriyage's avatar
      Adds INFO fields to track fork child progress (#8414) · fd052d2a
      uriyage authored
      * Adding current_save_keys_total and current_save_keys_processed info fields.
        Present in replication, BGSAVE and AOFRW.
      * Changing RM_SendChildCOWInfo() to RM_SendChildHeartbeat(double progress)
      * Adding new info field current_fork_perc. Present in Replication, BGSAVE, AOFRW,
        and module forks.
      fd052d2a
  7. 15 Feb, 2021 2 commits
  8. 09 Feb, 2021 1 commit
  9. 29 Jan, 2021 3 commits
    • filipe oliveira's avatar
      Enabled background and reply time tracking on blocked on keys/blocked on... · f0c5052a
      filipe oliveira authored
      Enabled background and reply time tracking on blocked on keys/blocked on background work clients (#7491)
      
      This commit enables tracking time of the background tasks and on replies,
      opening the door for properly tracking commands that rely on blocking / background
       work via the slowlog, latency history, and commandstats. 
      
      Some notes:
      - The time spent blocked waiting for key changes, or blocked on synchronous
        replication is not accounted for. 
      
      - **This commit does not affect latency tracking of commands that are non-blocking
        or do not have background work.** ( meaning that it all stays the same with exception to
        `BZPOPMIN`,`BZPOPMAX`,`BRPOP`,`BLPOP`, etc... and module's commands that rely
        on background threads ). 
      
      -  Specifically for latency history command we've added a new event class named
        `command-unblocking` that will enable latency monitoring on commands that spawn
        background threads to do the work.
      
      - For blocking commands we're now considering the total time of a command as the
        time spent on call() + the time spent on replying when unblocked.
      
      - For Modules commands that rely on background threads we're now considering the
        total time of a command as the time spent on call (main thread) + the time spent on
        the background thread ( if marked within `RedisModule_MeasureTimeStart()` and
        `RedisModule_MeasureTimeEnd()` ) + the time spent on replying (main thread)
      
      To test for this feature we've added a `unit/moduleapi/blockonbackground` test that relies on
      a module that blocks the client and sleeps on the background for a given time. 
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time
      - check blocked command that uses RedisModule_MeasureTimeStart() is tracking background time even in timeout
      - check blocked command with multiple calls RedisModule_MeasureTimeStart()  is tracking the total background time
      - check blocked command without calling RedisModule_MeasureTimeStart() is not reporting background time
      f0c5052a
    • Yang Bodong's avatar
      Add HRANDFIELD and ZRANDMEMBER. improvements to SRANDMEMBER (#8297) · b9a0500f
      Yang Bodong authored
      
      
      New commands:
      `HRANDFIELD [<count> [WITHVALUES]]`
      `ZRANDMEMBER [<count> [WITHSCORES]]`
      Algorithms are similar to the one in SRANDMEMBER.
      
      Both return a simple bulk response when no arguments are given, and an array otherwise.
      In case values/scores are requested, RESP2 returns a long array, and RESP3 a nested array.
      note: in all 3 commands, the only option that also provides random order is the one with negative count.
      
      Changes to SRANDMEMBER
      * Optimization when count is 1, we can use the more efficient algorithm of non-unique random
      * optimization: work with sds strings rather than robj
      
      Other changes:
      * zzlGetScore: when zset needs to convert string to double, we use safer memcpy (in
        case the buffer is too small)
      * Solve a "bug" in SRANDMEMBER test: it intended to test a positive count (case 3 or
        case 4) and by accident used a negative count
      Co-authored-by: default avatarxinluton <xinluton@qq.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b9a0500f
    • zhaozhao.zz's avatar
      AOF: recover from last write error after turn on appendonly again (#8030) · 49b36633
      zhaozhao.zz authored
      The key point is how to recover from last AOF write error, for example:
      
      1. start redis with appendonly yes, and append some write commands
      
      2. short write or something else error happen, `server.aof_last_write_status` changed to `C_ERR`, now redis doesn't accept write commands
      
      3. execute `CONFIG SET appendonly no` to avoid the above problem, now redis can accept write commands again
      
      4. disk error resolved, and execute `CONFIG SET appendonly yes` to reopen AOF, but `server.aof_last_write_status` cannot be changed to `C_OK` (if background aof rewrite run less then 1 second, it will free `server.aof_buf` and then serverCron cannot fix `aof_last_write_status`), then redis cannot accept write commands forever.
      
      This PR use a simple way to fix it:
      
      1. just free `server.aof_buf` when stop appendonly to save memory, if error happens in `flushAppendOnlyFile(1)`, the `server.aof_buf` may contains some data which has not be written to aof, I think we can ignore it because we turn off the appendonly.
      
      2. reset fsync status after stop appendonly and call `flushAppendOnlyFile` only when `aof_state` is ON
      
      3. reset `server.last_write_status` when reopen aof to accept write commands
      49b36633
  10. 28 Jan, 2021 5 commits
    • Allen Farris's avatar
      implement FAILOVER command (#8315) · 0d18a1e8
      Allen Farris authored
      Implement FAILOVER command, which coordinates failover
      between the server and one of its replicas.
      0d18a1e8
    • Yossi Gottlieb's avatar
      Add proc-title-template option. (#8397) · 4bb5ccbe
      Yossi Gottlieb authored
      Make it possible to customize the process title, i.e. include custom
      strings, immutable configuration like port, tls-port, unix socket name,
      etc.
      4bb5ccbe
    • guybe7's avatar
      Modules: Add event for fork child birth and termination (#8289) · 01cbf17b
      guybe7 authored
      Useful to avoid doing background jobs that can cause excessive COW
      01cbf17b
    • Z. Liu's avatar
      Add 'set-proc-title' config so that this mechanism can be disabled (#3623) · 17b34c73
      Z. Liu authored
      
      
      if option `set-proc-title' is no, then do nothing for proc title.
      
      The reason has been explained long ago, see following:
      
      We update redis to 2.8.8, then found there are some side effect when
      redis always change the process title.
      
      We run several slave instance on one computer, and all these salves
      listen on unix socket only, then ps will show:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server *:0
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server *:0
      
      for redis 2.6 the output of ps is like following:
      
        1 S redis 18036 1 0 80 0 - 56130 ep_pol 14:02 ? 00:00:31 /usr/sbin/redis-server /etc/redis/a.conf
        1 S redis 23949 1 0 80 0 - 11074 ep_pol 15:41 ? 00:00:00 /usr/sbin/redis-server /etc/redis/b.conf
      
      Later is more informational in our case. The situation
      is worse when we manage the config and process running
      state by salt. Salt check the process by running "ps |
      grep SIG" (for Gentoo System) to check the running
      state, where SIG is the string to search for when
      looking for the service process with ps. Previously, we
      define sig as "/usr/sbin/redis-server
      /etc/redis/a.conf". Since the ps output is identical for
      our case, so we have no way to check the state of
      specified redis instance.
      
      So, for our case, we prefer the old behavior, i.e, do
      not change the process title for the main redis process.
      Or add an option such as "set-proc-title [yes|no]" to
      control this behavior.
      Co-authored-by: default avatarYossi Gottlieb <yossigo@gmail.com>
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      17b34c73
    • Yossi Gottlieb's avatar
      Avoid assertions when testing arm64 cow bug. (#8405) · 3a504904
      Yossi Gottlieb authored
      At least in one case the arm64 cow kernel bug test triggers an assert, which is a problem because it cannot be ignored like cases where the bug is found.
      
      On older systems (Linux <4.5) madvise fails because MADV_FREE is not supported. We treat these failures as an indication the system is not affected.
      
      Fixes #8351, #8406
      3a504904
  11. 27 Jan, 2021 1 commit
    • Raghav Muddur's avatar
      GETEX, GETDEL and SET PXAT/EXAT (#8327) · 0367a808
      Raghav Muddur authored
      This commit introduces two new command and two options for an existing command
      
      GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
      [PXAT milliseconds-timestamp]
      
      The getexCommand() function implements extended options and variants of the GET
      command. Unlike GET command this command is not read-only. Only one of the options
      can be used at a given time.
      
      1. PERSIST removes any TTL associated with the key.
      2. EX Set expiry TTL in seconds.
      3. PX Set expiry TTL in milliseconds.
      4. EXAT Same like EX instead of specifying the number of seconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      5. PXAT Same like PX instead of specifying the number of milliseconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      
      Command would return either the bulk string, error or nil.
      
      GETDEL <key>
      Would delete the key after getting.
      
      SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] [PX <milliseconds>]
      [EXAT <seconds-timestamp>][PXAT <milliseconds-timestamp>]
      
      Two new options added here are EXAT and PXAT
      
      Key implementation notes
      - `SET` with `PX/EX/EXAT/PXAT` is always translated to `PXAT` in `AOF`. When relative time is
        specified (`PX/EX`), replication will always use `PX`.
      - `setexCommand` and `psetexCommand` would no longer need translation in `feedAppendOnlyFile`
        as they are modified to invoke `setGenericCommand ` with appropriate flags which will take care of
        correct AOF translation.
      - `GETEX` without any optional argument behaves like `GET`.
      - `GETEX` command is never propagated, It is either propagated as `PEXPIRE[AT], or PERSIST`.
      - `GETDEL` command is propagated as `DEL`
      - Combined the validation for `SET` and `GETEX` arguments. 
      - Test cases to validate AOF/Replication propagation
      0367a808
  12. 26 Jan, 2021 1 commit
    • Wen Hui's avatar
      Sentinel: Fix Config Dependency and Rewrite Sequence (#8271) · 1aad55b6
      Wen Hui authored
      This commit fixes a well known and an annoying issue in Sentinel mode.
      
      Cause of this issue:
      Currently, Redis rewrite process works well in server mode, however in sentinel mode,
      the sentinel config has variant semantics for different configurations, in example configuration
      https://github.com/redis/redis/blob/unstable/sentinel.conf, we put comments on these.
      However the rewrite process only treat the sentinel config as a single option. During rewrite
      process, it will mess up with the lines and comments.
      
      Approaches:
      In order to solve this issue, we need to differentiate different subconfig options in sentinel separately,
      for example, sentinel monitor <master-name> <ip> <redis-port> <quorum>
      we can treat it as sentinel monitor option, instead of the sentinel option.
      
      This commit also fixes the dependency issue when putting configurations in sentinel.conf.
      For example before this commit,we must put
      `sentinel monitor <master-name> <ip> <redis-port> <quorum>` before
      `sentinel auth-pass <master-name> <password>` for a single master,
      otherwise the server cannot start and will return error. This commit fixes this issue, as long as
      the monitoring master was configured, no matter the sequence is, the sentinel can start and run properly.
      1aad55b6
  13. 20 Jan, 2021 3 commits
    • Yossi Gottlieb's avatar
      CONFIG REWRITE should honor umask settings. (#8371) · b548ffab
      Yossi Gottlieb authored
      Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one.
      
      The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly.
      
      Fixes #8369
      b548ffab
    • Guy Korland's avatar
      Add CI for FreeBSD (#8292) · ac5f21d6
      Guy Korland authored
      
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ac5f21d6
    • guybe7's avatar
      Fix firstkey,lastkey,step in COMMAND command for some commands (#8367) · baf92f3f
      guybe7 authored
      The output for COMMAND command was wrong for some commands.
      clients can use firstkey,lastkey,step to find (some) key name arguments, and the
      "movablekeys" flag to know that they can't know all (or any) of the key name arguments.
      
      These commands had the wrong output:
      1. GEORADIUS*_RO used to have "movablekeys" (which it doesn't really need)
      2. XREAD and XREADGROUP used to have (1,1,1). but that's completely wrong.
      3. Z*STORE used to have (0,0,0) but it can at lest give the index of the dstkey (1,1,1)
      baf92f3f
  14. 19 Jan, 2021 1 commit
  15. 13 Jan, 2021 1 commit
  16. 12 Jan, 2021 1 commit
    • Madelyn Olson's avatar
      Fix issues in wait test (#8310) · b24b4903
      Madelyn Olson authored
      This fixes three issues:
      1.  Using debug SLEEP was impacting the subsequent test, and causing it to pass reliably even though it should have failed. There was exactly 5 seconds of artificial pause (after 1000, wait 3000, wait 1000) between the debug sleep 5 and when we needed to unblock the client in the subsequent test. Now the test properly makes sure the client is unblocked, and the subsequent test is fixed.
      2. Minor, the client pause types were using & comparisons instead of ==, since it was previously a flag.
      3. Test is faster now that some of the hand wavy time is removed. 
      b24b4903
  17. 08 Jan, 2021 1 commit
  18. 07 Jan, 2021 4 commits
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · b02780c4
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b02780c4
    • YaacovHazan's avatar
      Report child copy-on-write info continuously · ea930a35
      YaacovHazan authored
      Add INFO field, rdb_active_cow_size, to report COW of a live fork child while
      it's active.
      - once in 1024 keys check the time, and if there's more than one second since
        the last report send a report to the parent via the pipe.
      - refactor the child_info_data struct, it's an implementation detail that
        shouldn't be in the server struct, and not used to communicate data between
        caller and callee
      - remove the magic value from that struct (not sure what it was good for), and
        instead add handling of short reads.
      - add another value to the structure, cow_type, to indicate if the report is
        for the new rdb_active_cow_size field, or it's the last report of a
        successful operation
      - add new Module API to report the active COW
      - add more asserts variants to test.tcl
      ea930a35
    • YaacovHazan's avatar
      Refactory fork child related infra, Unify child pid · f9dacf8a
      YaacovHazan authored
      This is a refactory commit, isn't suppose to have any actual impact.
      it does the following:
      - keep just one server struct fork child pid variable instead of 3
      - have one server struct variable indicating the purpose of the current fork
        child.
      - redisFork is now responsible of updating the server struct with the pid,
        which means it can be the one that calls updateDictResizePolicy
      - move child info pipe handling into redisFork instead of having them
        repeated outside
      - there are two classes of fork purposes, mutually exclusive group (AOF, RDB,
        Module), and one that can create several forks to coexist in parallel (LDB,
        but maybe Modules some day too, Module API allows for that).
      - minor fix to killRDBChild:
        unlike killAppendOnlyChild and TerminateModuleForkChild, the killRDBChild
        doesn't clear the pid variable or call wait4, so checkChildrenDone does
        the cleanup for it.
        This commit removes the explicit calls to rdbRemoveTempFile, closeChildInfoPipe,
        updateDictResizePolicy, which didn't do any harm, but where unnecessary.
      f9dacf8a
    • Jonah H. Harris's avatar
      Add ZRANGESTORE command, and improve ZSTORE command (#7844) · b5029dfd
      Jonah H. Harris authored
      
      
      Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX].
      
      Syntax for the new ZRANGESTORE command:
      ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count]
      
      New syntax for ZRANGE:
      ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count]
      
      Old syntax for ZRANGE:
      ZRANGE [WITHSCORES]
      
      Other ZRANGE commands remain unchanged.
      
      The implementation uses common code for all of these, by utilizing a consumer interface that in one
      command response to the client, and in the other command stores a zset key.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b5029dfd
  19. 06 Jan, 2021 1 commit
    • guybe7's avatar
      Add XAUTOCLAIM (#7973) · 714e103a
      guybe7 authored
      
      
      New command: XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT <count>] [JUSTID]
      
      The purpose is to claim entries from a stale consumer without the usual
      XPENDING+XCLAIM combo which takes two round trips.
      
      The syntax for XAUTOCLAIM is similar to scan: A cursor is returned (streamID)
      by each call and should be used as start for the next call. 0-0 means the scan is complete.
      
      This PR extends the deferred reply mechanism for any bulk string (not just counts)
      
      This PR carries some unrelated test code changes:
      - Renames the term "client" into "consumer" in the stream-cgroups test
      - And also changes DEBUG SLEEP into "after"
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      714e103a
  20. 04 Jan, 2021 2 commits
    • Itamar Haber's avatar
      HELP subcommand, continued (#5531) · 9dcdc7e7
      Itamar Haber authored
      
      
      * man-like consistent long formatting
      * Uppercases commands, subcommands and options
      * Adds 'HELP' to HELP for all
      * Lexicographical order
      * Uses value notation and other .md likeness
      * Moves const char *help to top
      * Keeps it under 80 chars
      * Misc help typos, consistent conjuctioning (i.e return and not returns)
      * Uses addReplySubcommandSyntaxError(c) all over
      Signed-off-by: default avatarItamar Haber <itamar@redislabs.com>
      9dcdc7e7
    • Meir Shpilraien (Spielrein)'s avatar
      Fix assertion on loading AOF with timed out script. (#8284) · ecd53518
      Meir Shpilraien (Spielrein) authored
      If AOF file contains a long Lua script that timed out, then the `evalCommand` calls
      `blockingOperationEnds` which sets `server.blocked_last_cron` to 0. later on,
      the AOF `whileBlockedCron` function asserts that this value is not 0.
      
      The fix allows nesting call to `blockingOperationStarts` and `blockingOperationEnds`.
      
      The issue was first introduce in this commit: 9ef8d2f6 (Redis 6.2 RC1)
      ecd53518
  21. 03 Jan, 2021 1 commit
    • Oran Agra's avatar
      Fix rare assertion as a result of: active defrag while loading (#8281) · 7896debe
      Oran Agra authored
      In #7726 (part of 6.2), we added a mechanism for whileBlockedCron, this
      mechanism has an assertion to make sure the timestamp in
      whileBlockedCron was always set correctly before the blocking operation
      starts.
      
      I now found (thanks to our CI) two bugs in that area:
      1) CONFIG RESETSTAT (if it was allowed during loading) would have
         cleared this var
      2) the call stopLoading (which calls whileBlockedCron) was made too
         early, while the rio is still in use, in which case the update_cksum
         (rdbLoadProgressCallback) may still be called and whileBlockedCron
         can assert.
      7896debe
  22. 02 Jan, 2021 1 commit
  23. 31 Dec, 2020 1 commit
    • filipe oliveira's avatar
      Add errorstats info section, Add failed_calls and rejected_calls to commandstats (#8217) · 90b9f08e
      filipe oliveira authored
      This Commit pushes forward the observability on overall error statistics and command statistics within redis-server:
      
      It extends INFO COMMANDSTATS to have
      - failed_calls in - so we can keep track of errors that happen from the command itself, broken by command.
      - rejected_calls - so we can keep track of errors that were triggered outside the commmand processing per se
      
      Adds a new section to INFO, named ERRORSTATS that enables keeping track of the different errors that
      occur within redis ( within processCommand and call ) based on the reply Error Prefix ( The first word
      after the "-", up to the first space ).
      
      This commit also fixes RM_ReplyWithError so that it can be correctly identified as an error reply.
      90b9f08e
  24. 27 Dec, 2020 1 commit
    • Oran Agra's avatar
      Fix memory leaks in error replies due to recent change (#8249) · 049cf8cd
      Oran Agra authored
      Recently efaf09ee started using addReplyErrorSds in place of
      addReplySds the later takes ownership of the string but the former did
      not.
      This introduced memory leaks when a script returns an error to redis,
      and also in clusterRedirectClient (two new usages of
      addReplyErrorSds which was mostly unused till now.
      
      This commit chagnes two thanks.
      1. change addReplyErrorSds to take ownership of the error string.
      2. scripting.c doesn't actually need to use addReplyErrorSds, it's a
      perfect match for addReplyErrorFormat (replaces newlines with spaces)
      049cf8cd
  25. 25 Dec, 2020 1 commit
    • Itamar Haber's avatar
      Adds count to L/RPOP (#8179) · f44186e5
      Itamar Haber authored
      Adds: `L/RPOP <key> [count]`
      
      Implements no. 2 of the following strategies:
      
      1. Loop on listTypePop - this would result in multiple calls for memory freeing and allocating (see https://github.com/redis/redis/pull/8179/commits/769167a079b0e110d28e4a8099dce1ecd45682b5)
      2. Iterate the range to build the reply, then call quickListDelRange - this requires two iterations and **is the current choice**
      3. Refactor quicklist to have a pop variant of quickListDelRange - probably optimal but more complex
      
      Also:
      * There's a historical check for NULL after calling listTypePop that was converted to an assert.
      * This refactors common logic shared between LRANGE and the new form of LPOP/RPOP into addListRangeReply (adds test for b/w compat)
      * Consequently, it may have made sense to have `LRANGE l -1 -2` and `LRANGE l 9 0` be legit and return a reverse reply. Due to historical reasons that would be, however, a breaking change.
      * Added minimal comments to existing commands to adhere to the style, make core dev life easier and get commit karma, naturally.
      f44186e5
  26. 24 Dec, 2020 2 commits