1. 26 Jan, 2021 3 commits
    • Wen Hui's avatar
      Sentinel: Fix Config Dependency and Rewrite Sequence (#8271) · 1aad55b6
      Wen Hui authored
      This commit fixes a well known and an annoying issue in Sentinel mode.
      
      Cause of this issue:
      Currently, Redis rewrite process works well in server mode, however in sentinel mode,
      the sentinel config has variant semantics for different configurations, in example configuration
      https://github.com/redis/redis/blob/unstable/sentinel.conf, we put comments on these.
      However the rewrite process only treat the sentinel config as a single option. During rewrite
      process, it will mess up with the lines and comments.
      
      Approaches:
      In order to solve this issue, we need to differentiate different subconfig options in sentinel separately,
      for example, sentinel monitor <master-name> <ip> <redis-port> <quorum>
      we can treat it as sentinel monitor option, instead of the sentinel option.
      
      This commit also fixes the dependency issue when putting configurations in sentinel.conf.
      For example before this commit,we must put
      `sentinel monitor <master-name> <ip> <redis-port> <quorum>` before
      `sentinel auth-pass <master-name> <password>` for a single master,
      otherwise the server cannot start and will return error. This commit fixes this issue, as long as
      the monitoring master was configured, no matter the sequence is, the sentinel can start and run properly.
      1aad55b6
    • Vladimir Maksimovski's avatar
      Remove check leading to duplicate branches (#8398) · 70789cf4
      Vladimir Maksimovski authored
      Remove check leading to duplicate branches and unused withscores parameter
      70789cf4
    • zhaozhao.zz's avatar
      latency: add expire-del event (#8394) · 951d25ff
      zhaozhao.zz authored
      951d25ff
  2. 22 Jan, 2021 1 commit
  3. 21 Jan, 2021 3 commits
  4. 20 Jan, 2021 4 commits
    • Yossi Gottlieb's avatar
      CONFIG REWRITE should honor umask settings. (#8371) · b548ffab
      Yossi Gottlieb authored
      Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one.
      
      The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly.
      
      Fixes #8369
      b548ffab
    • Guy Korland's avatar
      Add CI for FreeBSD (#8292) · ac5f21d6
      Guy Korland authored
      
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      ac5f21d6
    • guybe7's avatar
      Fix firstkey,lastkey,step in COMMAND command for some commands (#8367) · baf92f3f
      guybe7 authored
      The output for COMMAND command was wrong for some commands.
      clients can use firstkey,lastkey,step to find (some) key name arguments, and the
      "movablekeys" flag to know that they can't know all (or any) of the key name arguments.
      
      These commands had the wrong output:
      1. GEORADIUS*_RO used to have "movablekeys" (which it doesn't really need)
      2. XREAD and XREADGROUP used to have (1,1,1). but that's completely wrong.
      3. Z*STORE used to have (0,0,0) but it can at lest give the index of the dstkey (1,1,1)
      baf92f3f
    • Viktor Söderqvist's avatar
      More modules API ref formatting fixes (#8344) · 16258f21
      Viktor Söderqvist authored
      Fix broken formatting in `RM_Call` and `RM_CreateDataType`,
      `RM_SubscribeToServerEvent` (nested lists, etc. in list items).
      
      Unhide docs of `RM_LoadDataTypeFromString` and
      `RM_SaveDataTypeToString` by removing blank line between docs and
      function.
      
      Clarification added to `RM__Assert`: Recommentation to use the
      `RedisModule_Assert` macro instead.
      
      All names containing underscores (variable and macro names) are
      wrapped in backticks (if not already wrapped in backticks). This
      prevents underscore from being interpreted as italics in some
      cases.
      
      Names including a wildcard star, e.g. RM_Defrag*(), is wrapped in
      backticks (and RM replaced by RedisModule in this case). This
      prevents the * from being interpreted as an italics marker.
      
      A list item with a sublist, a paragraph and another sublist is a
      combination which seems impossible to achieve with RedCarped
      markdown, so the one occurrence of this is rewritten.
      
      Various trivial changes (typos, b...
      16258f21
  5. 19 Jan, 2021 4 commits
  6. 18 Jan, 2021 1 commit
  7. 17 Jan, 2021 1 commit
  8. 15 Jan, 2021 3 commits
    • Yang Bodong's avatar
      Add lazyfree-lazy-user-flush config to control default behavior of... · 294f93af
      Yang Bodong authored
      Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH (#8258)
      
      * Adds ASYNC and SYNC arguments to SCRIPT FLUSH
      * Adds SYNC argument to FLUSHDB and FLUSHALL
      * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH.
      
      the new behavior is as follows:
      * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the
        value of lazyfree-lazy-user-flush.
      * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner.
      * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
      294f93af
    • Viktor Söderqvist's avatar
      Rename non-API RM-prefixed functions to hide them from API docs · fcb3dfe5
      Viktor Söderqvist authored
      The prefix is changed from `RM_` to `module` on the following
      internal functions, to prevent them from appearing in the API docs:
      
          RM_LogRaw -> moduleLogRaw
          RM_FreeCallReplyRec -> moduleFreeCallReplyRec
          RM_ZsetAddFlagsToCoreFlags -> moduleZsetAddFlagsToCoreFlags
          RM_ZsetAddFlagsFromCoreFlags -> moduleZsetAddFlagsFromCoreFlags
      fcb3dfe5
    • Viktor Söderqvist's avatar
      Modules API reference formatting fixes · ebf20b83
      Viktor Söderqvist authored
      Fixes markdown formatting errors and some functions not showing
      up in the generated documentation at all.
      
      Ruby script (gendoc.rb) fixes:
      
      * Modified automatic instertion of backquotes:
        * Don't add backquotes around names which are already preceded by a
          backquote. Fixes for example \`RedisModule_Reply\*\` which turning
          into \`\`RedisModule_Reply\`\*\` messes up the formatting.
        * Add backquotes around types such as RedisModuleString (in addition
          to function names `RedisModule_[A-z()]*` and macro names
          `REDISMODULE_[A-z]*`).
        * Require 4 spaces indentation for disabling automatic backquotes, i.e.
          code blocks. Fixes continuations of list items (indented 2 spaces).
      * More permissive extraction of doc comments:
        * Allow doc comments starting with `/**`.
        * Make space before `*` on each line optional.
        * Make space after `/*` and `/**` optional (needed when appearing on
          its own line).
      
      Markdown fixes in module.c:
      
      * Fix code blocks not indented enough (4 spaces needed).
      * Add black line before code blocks and lists where missing (needed).
      * Enclose special markdown characters `_*^<>` in backticks to prevent them
        from messing up formatting.
      * Lists with `1)` changed to `1.` for proper markdown lists.
      * Remove excessive indentation which causes text to be unintentionally
        rendered as code blocks.
      * Other minor formatting fixes.
      
      Other fixes in module.c:
      
      * Remove blank lines between doc comment and function definition. A blank
        line here makes the Ruby script exclude the function in docs.
      ebf20b83
  9. 14 Jan, 2021 1 commit
  10. 13 Jan, 2021 4 commits
    • Wang Yuan's avatar
      Optimize performance of clusterGenNodesDescription for large clusters (#8182) · 9cb9f98d
      Wang Yuan authored
      Optimize the performance of clusterGenNodesDescription by only checking slot ownership of each slot once, instead of checking each slot for each node. 
      9cb9f98d
    • houzj.fnst's avatar
      remove some unnecessary checks (#7431) · f5577fdb
      houzj.fnst authored
      Remove several checks that always evaluate to true.
      f5577fdb
    • sundb's avatar
      Fix use of lookupKeyRead and lookupKeyWrite in zrangeGenericCommand,... · 8a81ed1b
      sundb authored
      Fix use of  lookupKeyRead and lookupKeyWrite in zrangeGenericCommand, zunionInterDiffGenericCommand (#8316)
      
      * Change zunionInterDiffGenericCommand to use lookupKeyRead if dstkey is null
      * Change zrangeGenericCommand to use lookupKey Write if dstkey isn't null
      
      ZRANGESTORE and UNION, ZINTER, ZDIFF are all new commands (6.2 RC1 and RC2).
      In redis 6.0 the ZRANGE was using lookupKeyRead, and ZUNIONSTORE / ZINTERSTORE were using lookupKeyWrite.
      So there bugs are introduced in 6.2 and will be resolved before it is released.
      
      the implications of this bug are also not big:
      The sole difference between LookupKeyRead and LookupKeyWrite is for command executed on a replica, which are not received from its master client. (for the master, and for the master client on the replica, these two functions behave the same)!
      8a81ed1b
    • sundb's avatar
      Fix units in log message of copy-on-write (#8320) · 593cde16
      sundb authored
      593cde16
  11. 12 Jan, 2021 2 commits
    • Madelyn Olson's avatar
      Fix issues in wait test (#8310) · b24b4903
      Madelyn Olson authored
      This fixes three issues:
      1.  Using debug SLEEP was impacting the subsequent test, and causing it to pass reliably even though it should have failed. There was exactly 5 seconds of artificial pause (after 1000, wait 3000, wait 1000) between the debug sleep 5 and when we needed to unblock the client in the subsequent test. Now the test properly makes sure the client is unblocked, and the subsequent test is fixed.
      2. Minor, the client pause types were using & comparisons instead of ==, since it was previously a flag.
      3. Test is faster now that some of the hand wavy time is removed. 
      b24b4903
    • Bob Li's avatar
      Fix overflow of rdbWriteRaw return value (#8306) · 542455ce
      Bob Li authored
      Saving string of more than 2GB to the RDB file, can result in corrupt RDB, or failure in rdbSave.
      S
      542455ce
  12. 09 Jan, 2021 3 commits
  13. 08 Jan, 2021 5 commits
    • Oran Agra's avatar
      Fix last COW INFO report, Skip test on non-linux platforms (#8301) · 8dd16cae
      Oran Agra authored
      - the last COW report wasn't always read from the pipe
        (receiveLastChildInfo wasn't used)
      - but in fact, there's no reason we won't always try to drain that pipe
        so i'm unifying receiveLastChildInfo with receiveChildInfo
      - adjust threshold of the COW test when run in accurate mode
      - add some prints in case this test fails again
      - fix indentation, page size, and PID! in MacOS proc info
      
      p.s. it seems that pri_pages_dirtied is always 0
      8dd16cae
    • Yang Bodong's avatar
      GEOSEARCH - ANY option, for limited search that returns ASAP (#8259) · ea5350c5
      Yang Bodong authored
      Support ANY option to return some results that match the criteria ASAP,
      without a complete search and implicit sorting.
      ea5350c5
    • guybe7's avatar
      XADD and XTRIM, Trim by MINID, and new LIMIT argument (#8169) · 814aad65
      guybe7 authored
      This PR adds another trimming strategy to XADD and XTRIM named MINID
      (complements the existing MAXLEN).
      It also adds a new LIMIT argument that allows incremental trimming by repeated
      calls (rather than all at once).
      
      This provides the ability to trim all records older than a certain ID (which makes it
      possible for the user to trim by age too).
      Example:
      XTRIM mystream MINID ~ 1608540753 will trim entries with id < 1608540753,
      but might not trim all (because of the ~ modifier)
      
      The purpose is to ease the use of streams. many users use streams as logs and
      the common case is wanting a log
      of the last X seconds rather than a log that contains maximum X entries (new
      MINID vs existing MAXLEN)
      
      The new LIMIT modifier is only supported when the trim strategy uses ~.
      i.e. when the user asked for exact trimming, it all happens in one go (no
      possibility for incremental trimming).
      However, when ~ is provided, we trim full rax nodes, up to the limit number
      of records.
      The default limit is 100*stream_node_max_entries (used when LIMIT is not
      provided).
      I.e. this is a behavior change (even if the existing MAXLEN strategy is used).
      An explicit limit of 0 means unlimited (but note that it's not the default).
      
      Other changes:
      
      Refactor arg parsing code for XADD and XTRIM to use common code.
      814aad65
    • Madelyn Olson's avatar
      Throw error for conflicting bcast tracking prefixes (#8176) · 999494ce
      Madelyn Olson authored
      Throw an error if there are conflicting bcast tracking prefixes. 
      999494ce
    • Madelyn Olson's avatar
      Add support for client pause WRITE (#8170) · 47579bdf
      Madelyn Olson authored
      Implementation of client pause WRITE and client unpause
      47579bdf
  14. 07 Jan, 2021 4 commits
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · b02780c4
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b02780c4
    • YaacovHazan's avatar
      Report child copy-on-write info continuously · ea930a35
      YaacovHazan authored
      Add INFO field, rdb_active_cow_size, to report COW of a live fork child while
      it's active.
      - once in 1024 keys check the time, and if there's more than one second since
        the last report send a report to the parent via the pipe.
      - refactor the child_info_data struct, it's an implementation detail that
        shouldn't be in the server struct, and not used to communicate data between
        caller and callee
      - remove the magic value from that struct (not sure what it was good for), and
        instead add handling of short reads.
      - add another value to the structure, cow_type, to indicate if the report is
        for the new rdb_active_cow_size field, or it's the last report of a
        successful operation
      - add new Module API to report the active COW
      - add more asserts variants to test.tcl
      ea930a35
    • YaacovHazan's avatar
      Refactory fork child related infra, Unify child pid · f9dacf8a
      YaacovHazan authored
      This is a refactory commit, isn't suppose to have any actual impact.
      it does the following:
      - keep just one server struct fork child pid variable instead of 3
      - have one server struct variable indicating the purpose of the current fork
        child.
      - redisFork is now responsible of updating the server struct with the pid,
        which means it can be the one that calls updateDictResizePolicy
      - move child info pipe handling into redisFork instead of having them
        repeated outside
      - there are two classes of fork purposes, mutually exclusive group (AOF, RDB,
        Module), and one that can create several forks to coexist in parallel (LDB,
        but maybe Modules some day too, Module API allows for that).
      - minor fix to killRDBChild:
        unlike killAppendOnlyChild and TerminateModuleForkChild, the killRDBChild
        doesn't clear the pid variable or call wait4, so checkChildrenDone does
        the cleanup for it.
        This commit removes the explicit calls to rdbRemoveTempFile, closeChildInfoPipe,
        updateDictResizePolicy, which didn't do any harm, but where unnecessary.
      f9dacf8a
    • Jonah H. Harris's avatar
      Add ZRANGESTORE command, and improve ZSTORE command (#7844) · b5029dfd
      Jonah H. Harris authored
      
      
      Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX].
      
      Syntax for the new ZRANGESTORE command:
      ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count]
      
      New syntax for ZRANGE:
      ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count]
      
      Old syntax for ZRANGE:
      ZRANGE [WITHSCORES]
      
      Other ZRANGE commands remain unchanged.
      
      The implementation uses common code for all of these, by utilizing a consumer interface that in one
      command response to the client, and in the other command stores a zset key.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b5029dfd
  15. 06 Jan, 2021 1 commit