1. 09 Jan, 2021 3 commits
  2. 08 Jan, 2021 6 commits
    • Oran Agra's avatar
      Fix last COW INFO report, Skip test on non-linux platforms (#8301) · 8dd16cae
      Oran Agra authored
      - the last COW report wasn't always read from the pipe
        (receiveLastChildInfo wasn't used)
      - but in fact, there's no reason we won't always try to drain that pipe
        so i'm unifying receiveLastChildInfo with receiveChildInfo
      - adjust threshold of the COW test when run in accurate mode
      - add some prints in case this test fails again
      - fix indentation, page size, and PID! in MacOS proc info
      
      p.s. it seems that pri_pages_dirtied is always 0
      8dd16cae
    • Yang Bodong's avatar
      GEOSEARCH - ANY option, for limited search that returns ASAP (#8259) · ea5350c5
      Yang Bodong authored
      Support ANY option to return some results that match the criteria ASAP,
      without a complete search and implicit sorting.
      ea5350c5
    • guybe7's avatar
      XADD and XTRIM, Trim by MINID, and new LIMIT argument (#8169) · 814aad65
      guybe7 authored
      This PR adds another trimming strategy to XADD and XTRIM named MINID
      (complements the existing MAXLEN).
      It also adds a new LIMIT argument that allows incremental trimming by repeated
      calls (rather than all at once).
      
      This provides the ability to trim all records older than a certain ID (which makes it
      possible for the user to trim by age too).
      Example:
      XTRIM mystream MINID ~ 1608540753 will trim entries with id < 1608540753,
      but might not trim all (because of the ~ modifier)
      
      The purpose is to ease the use of streams. many users use streams as logs and
      the common case is wanting a log
      of the last X seconds rather than a log that contains maximum X entries (new
      MINID vs existing MAXLEN)
      
      The new LIMIT modifier is only supported when the trim strategy uses ~.
      i.e. when the user asked for exact trimming, it all happens in one go (no
      possibility for incremental trimming).
      However, when ~ is provided, we trim full rax nodes, up to the limit number
      of records.
      The default limit is 100*stream_node_max_entries (used when LIMIT is not
      provided).
      I.e. this is a behavior change (even if the existing MAXLEN strategy is used).
      An explicit limit of 0 means unlimited (but note that it's not the default).
      
      Other changes:
      
      Refactor arg parsing code for XADD and XTRIM to use common code.
      814aad65
    • Oran Agra's avatar
      Skip defrag tests on systems with bigger page sizes (#8294) · 5843a45d
      Oran Agra authored
      The defragger works well on these systems, but the tests and their
      thresholds are not adjusted for these big pages, so the defragger isn't
      able to get down the fragmentation to the levels the test expects and it
      fails on "defrag didn't stop".
      
      Randomly choosing 8k as the threshold for the skipping
      
      Fixes #8265 (which had 65k pages)
      5843a45d
    • Madelyn Olson's avatar
      Throw error for conflicting bcast tracking prefixes (#8176) · 999494ce
      Madelyn Olson authored
      Throw an error if there are conflicting bcast tracking prefixes. 
      999494ce
    • Madelyn Olson's avatar
      Add support for client pause WRITE (#8170) · 47579bdf
      Madelyn Olson authored
      Implementation of client pause WRITE and client unpause
      47579bdf
  3. 07 Jan, 2021 4 commits
    • George Prekas's avatar
      Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224) · b02780c4
      George Prekas authored
      
      
      Older arm64 Linux kernels have a bug that could lead to data corruption during
      background save under the following scenario:
      
      1) jemalloc uses MADV_FREE on a page,
      2) jemalloc reuses and writes the page,
      3) Redis forks the background save process, and
      4) Linux performs page reclamation.
      
      Under these conditions, Linux will reclaim the page wrongfully and the
      background save process will read zeros when it tries to read the page.
      
      The bug has been fixed in Linux with commit:
      ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is
      preserved across pte_wrprotect()")
      
      This Commit adds an ignore-warnings config, when not found, redis will
      print a warning and exit on startup (default behavior).
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b02780c4
    • YaacovHazan's avatar
      Report child copy-on-write info continuously · ea930a35
      YaacovHazan authored
      Add INFO field, rdb_active_cow_size, to report COW of a live fork child while
      it's active.
      - once in 1024 keys check the time, and if there's more than one second since
        the last report send a report to the parent via the pipe.
      - refactor the child_info_data struct, it's an implementation detail that
        shouldn't be in the server struct, and not used to communicate data between
        caller and callee
      - remove the magic value from that struct (not sure what it was good for), and
        instead add handling of short reads.
      - add another value to the structure, cow_type, to indicate if the report is
        for the new rdb_active_cow_size field, or it's the last report of a
        successful operation
      - add new Module API to report the active COW
      - add more asserts variants to test.tcl
      ea930a35
    • YaacovHazan's avatar
      Refactory fork child related infra, Unify child pid · f9dacf8a
      YaacovHazan authored
      This is a refactory commit, isn't suppose to have any actual impact.
      it does the following:
      - keep just one server struct fork child pid variable instead of 3
      - have one server struct variable indicating the purpose of the current fork
        child.
      - redisFork is now responsible of updating the server struct with the pid,
        which means it can be the one that calls updateDictResizePolicy
      - move child info pipe handling into redisFork instead of having them
        repeated outside
      - there are two classes of fork purposes, mutually exclusive group (AOF, RDB,
        Module), and one that can create several forks to coexist in parallel (LDB,
        but maybe Modules some day too, Module API allows for that).
      - minor fix to killRDBChild:
        unlike killAppendOnlyChild and TerminateModuleForkChild, the killRDBChild
        doesn't clear the pid variable or call wait4, so checkChildrenDone does
        the cleanup for it.
        This commit removes the explicit calls to rdbRemoveTempFile, closeChildInfoPipe,
        updateDictResizePolicy, which didn't do any harm, but where unnecessary.
      f9dacf8a
    • Jonah H. Harris's avatar
      Add ZRANGESTORE command, and improve ZSTORE command (#7844) · b5029dfd
      Jonah H. Harris authored
      
      
      Add ZRANGESTORE command, and improve ZSTORE command to deprecated Z[REV]RANGE[BYSCORE|BYLEX].
      
      Syntax for the new ZRANGESTORE command:
      ZRANGESTORE [BYSCORE | BYLEX] [REV] [LIMIT offset count]
      
      New syntax for ZRANGE:
      ZRANGE [BYSCORE | BYLEX] [REV] [WITHSCORES] [LIMIT offset count]
      
      Old syntax for ZRANGE:
      ZRANGE [WITHSCORES]
      
      Other ZRANGE commands remain unchanged.
      
      The implementation uses common code for all of these, by utilizing a consumer interface that in one
      command response to the client, and in the other command stores a zset key.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      b5029dfd
  4. 06 Jan, 2021 2 commits
    • Wen Hui's avatar
      fix memory leak in processInlineBuffer error handling code (#8295) · cfcd0fa6
      Wen Hui authored
      This code path is normally executed only when v6.0 and above replicates from v2.4
      cfcd0fa6
    • guybe7's avatar
      Add XAUTOCLAIM (#7973) · 714e103a
      guybe7 authored
      
      
      New command: XAUTOCLAIM <key> <group> <consumer> <min-idle-time> <start> [COUNT <count>] [JUSTID]
      
      The purpose is to claim entries from a stale consumer without the usual
      XPENDING+XCLAIM combo which takes two round trips.
      
      The syntax for XAUTOCLAIM is similar to scan: A cursor is returned (streamID)
      by each call and should be used as start for the next call. 0-0 means the scan is complete.
      
      This PR extends the deferred reply mechanism for any bulk string (not just counts)
      
      This PR carries some unrelated test code changes:
      - Renames the term "client" into "consumer" in the stream-cgroups test
      - And also changes DEBUG SLEEP into "after"
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      714e103a
  5. 05 Jan, 2021 3 commits
    • huangzhw's avatar
      sdscatfmt call sdsMakeRoomFor, asked for more space than intended (#8286) · 595ecd5f
      huangzhw authored
      instead of asking for the extra new space it wanted, it asked to grow the
      string by the size it already has too.
      i.e. a string of 1000 bytes, needing to grow by 10 bytes, would have been
      asking for an **additional** 1010 bytes.
      595ecd5f
    • Oran Agra's avatar
      Fix rdb checksum / crc64 on bigendian (#8270) · 324070c8
      Oran Agra authored
      Turns out the RDB checksum in Redis 6.0 on bigendian is broken.
      It always returned 0, so the RDB files are generated as if checksum is
      disabled, and will be loaded ok on littleendian, and on bigendian.
      But it'll not be able to load RDB files generated on littleendian or older versions.
      
      Similarly DUMP and RESTORE will work on the same version (0==0),
      but will be unable to exchange dump payloads with littleendian or old versions.
      324070c8
    • Oran Agra's avatar
      Fix wrong order of key/value in Lua map response (#8266) · 2017407b
      Oran Agra authored
      When a Lua script returns a map to redis (a feature which was added in
      redis 6 together with RESP3), it would have returned the value first and
      the key second.
      
      If the client was using RESP2, it was getting them out of order, and if
      the client was in RESP3, it was getting a map of value => key.
      This was happening regardless of the Lua script using redis.setresp(3)
      or not.
      
      This also affects a case where the script was returning a map which it got
      from from redis by doing something like: redis.setresp(3); return redis.call()
      
      This fix is a breaking change for redis 6.0 users who happened to rely
      on the wrong order (either ones that used redis.setresp(3), or ones that
      returned a map explicitly).
      
      This commit also includes other two changes in the tests:
      1. The test suite now handles RESP3 maps as dicts rather than nested
         lists
      2. Remove some redundant (duplicate) tests from tracking.tcl
      2017407b
  6. 04 Jan, 2021 5 commits
    • Oran Agra's avatar
      df9c213b
    • Itamar Haber's avatar
      HELP subcommand, continued (#5531) · 9dcdc7e7
      Itamar Haber authored
      
      
      * man-like consistent long formatting
      * Uppercases commands, subcommands and options
      * Adds 'HELP' to HELP for all
      * Lexicographical order
      * Uses value notation and other .md likeness
      * Moves const char *help to top
      * Keeps it under 80 chars
      * Misc help typos, consistent conjuctioning (i.e return and not returns)
      * Uses addReplySubcommandSyntaxError(c) all over
      Signed-off-by: default avatarItamar Haber <itamar@redislabs.com>
      9dcdc7e7
    • Yang Bodong's avatar
      Swapdb should make transaction fail if there is any client watching keys (#8239) · 10f94b0a
      Yang Bodong authored
      
      
      This PR not only fixes the problem that swapdb does not make the
      transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to
      set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients.
      
      FLUSHDB was changed to first iterate on all watched keys, and then on the
      clients watching each key.
      Instead of iterating though all clients, and for each iterate on watched keys.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      10f94b0a
    • Meir Shpilraien (Spielrein)'s avatar
      Fix assertion on loading AOF with timed out script. (#8284) · ecd53518
      Meir Shpilraien (Spielrein) authored
      If AOF file contains a long Lua script that timed out, then the `evalCommand` calls
      `blockingOperationEnds` which sets `server.blocked_last_cron` to 0. later on,
      the AOF `whileBlockedCron` function asserts that this value is not 0.
      
      The fix allows nesting call to `blockingOperationStarts` and `blockingOperationEnds`.
      
      The issue was first introduce in this commit: 9ef8d2f6 (Redis 6.2 RC1)
      ecd53518
    • huangzhw's avatar
      sort Command lookupKeyRead and lookupKeyWrite are used on the opposite (#8283) · 08ad6abd
      huangzhw authored
      This is a recent problem, introduced by 74717438 (redis 6.0)
      
      The implications are:
      The sole difference between LookupKeyRead and LookupKeyWrite is for command
      executed on a replica, which are not received from its master client. (for the master,
      and for the master client on the replica, these two functions behave the same)!
      
      Since SORT is a write command, this bug only implicates a writable-replica.
      And these are its implications:
      
      - SORT STORE will behave as it did before the above mentioned commit (like before
        redis 6.0). on a writable-replica an already logically expired the key would have
        appeared missing. (store dest key would be deleted, instead of being populated
        with the data from the already logically expired key)
      - SORT (the non store variant, which in theory could have been executed on
        read-only-replica if it weren't for the write flag), will (in redis 6.0) have a new bug
        and return the data from the already logically expired key instead of empty response.
      08ad6abd
  7. 03 Jan, 2021 4 commits
  8. 02 Jan, 2021 1 commit
  9. 01 Jan, 2021 1 commit
  10. 31 Dec, 2020 1 commit
    • filipe oliveira's avatar
      Add errorstats info section, Add failed_calls and rejected_calls to commandstats (#8217) · 90b9f08e
      filipe oliveira authored
      This Commit pushes forward the observability on overall error statistics and command statistics within redis-server:
      
      It extends INFO COMMANDSTATS to have
      - failed_calls in - so we can keep track of errors that happen from the command itself, broken by command.
      - rejected_calls - so we can keep track of errors that were triggered outside the commmand processing per se
      
      Adds a new section to INFO, named ERRORSTATS that enables keeping track of the different errors that
      occur within redis ( within processCommand and call ) based on the reply Error Prefix ( The first word
      after the "-", up to the first space ).
      
      This commit also fixes RM_ReplyWithError so that it can be correctly identified as an error reply.
      90b9f08e
  11. 30 Dec, 2020 1 commit
  12. 27 Dec, 2020 3 commits
  13. 25 Dec, 2020 3 commits
    • Itamar Haber's avatar
      Adds count to L/RPOP (#8179) · f44186e5
      Itamar Haber authored
      Adds: `L/RPOP <key> [count]`
      
      Implements no. 2 of the following strategies:
      
      1. Loop on listTypePop - this would result in multiple calls for memory freeing and allocating (see https://github.com/redis/redis/pull/8179/commits/769167a079b0e110d28e4a8099dce1ecd45682b5)
      2. Iterate the range to build the reply, then call quickListDelRange - this requires two iterations and **is the current choice**
      3. Refactor quicklist to have a pop variant of quickListDelRange - probably optimal but more complex
      
      Also:
      * There's a historical check for NULL after calling listTypePop that was converted to an assert.
      * This refactors common logic shared between LRANGE and the new form of LPOP/RPOP into addListRangeReply (adds test for b/w compat)
      * Consequently, it may have made sense to have `LRANGE l -1 -2` and `LRANGE l 9 0` be legit and return a reverse reply. Due to historical reasons that would be, however, a breaking change.
      * Added minimal comments to existing commands to adhere to the style, make core dev life easier and get commit karma, naturally.
      f44186e5
    • Itamar Haber's avatar
      e18068d9
    • xhe's avatar
      fix the format · e6c1aeaf
      xhe authored
      
      Signed-off-by: default avatarxhe <xw897002528@gmail.com>
      e6c1aeaf
  14. 24 Dec, 2020 3 commits