1. 07 Jun, 2022 3 commits
    • Bjorn Svensson's avatar
      Documentation fixes of `BITFIELD_RO` and `XINFO STREAM` (#10823) · 1067cfb3
      Bjorn Svensson authored
      Correcting the introduction version of the command `BITFIELD_RO`
      Command added by commit: b3e4abf0 
      
      Add history info of the `FULL` modifier in `XINFO STREAM`
      Modifier was added by commit: 1e2aee39
      
      (Includes output from `./utils/generate-command-code.py`)
      1067cfb3
    • Binbin's avatar
      Handle multiple_token flag in generate-command-help.rb (#10822) · 3d56f607
      Binbin authored
      Currently generate-command.help.rb dose not handle the
      multiple_token flag, handle this flag in this PR.
      The format is the same as redis-cli rendering.
      ```diff
      - bitfield_ro key GET encoding offset [encoding offset ...]
      + bitfield_ro key GET encoding offset [GET encoding offset ...]
      ```
      
      Re run generate-command-code.py which was forget in #10820.
      Also change the flag value from string to bool, like "true" to true
      3d56f607
    • ranshid's avatar
      Fix bitfield_ro documentation (#10820) · b4772f63
      ranshid authored
      Current documentation only states a single GET token is needed which is not true. Marking the command with multiple_token.
      b4772f63
  2. 06 Jun, 2022 4 commits
    • Oran Agra's avatar
      update help.h (#10818) · b0272252
      Oran Agra authored
      before releasing 7.0.1
      b0272252
    • Oran Agra's avatar
      crash report instructions (#10816) · 475563e2
      Oran Agra authored
      Trying to avoid people opening crash report issues about module crashes and ARM QEMU bugs.
      475563e2
    • DarrenJiang13's avatar
      Split instantaneous_repl_total_kbps to instantaneous_input_repl_kbps and... · f5585834
      DarrenJiang13 authored
      Split instantaneous_repl_total_kbps to instantaneous_input_repl_kbps and instantaneous_output_repl_kbps. (#10810)
      
      A supplement to https://github.com/redis/redis/pull/10062
      Split `instantaneous_repl_total_kbps` to `instantaneous_input_repl_kbps` and `instantaneous_output_repl_kbps`. 
      ## Work:
      This PR:
      - delete 1 info field:
          - `instantaneous_repl_total_kbps`
      - add 2 info fields:
          - `instantaneous_input_repl_kbps / instantaneous_output_repl_kbps`
      ## Result:
      - master
      ```
      total_net_input_bytes:26633673
      total_net_output_bytes:21716596
      total_net_repl_input_bytes:0
      total_net_repl_output_bytes:18433052
      instantaneous_input_kbps:0.02
      instantaneous_output_kbps:0.00
      instantaneous_input_repl_kbps:0.00
      instantaneous_output_repl_kbps:0.00
      ```
      - slave
      ```
      total_net_input_bytes:18433212
      total_net_output_bytes:94790
      total_net_repl_input_bytes:18433052
      total_net_repl_output_bytes:0
      instantaneous_input_kbps:0.00
      instantaneous_output_kbps:0.05
      instantaneous_input_repl_kbps:0.00
      instantaneous_output_repl_kbps:0.00
      ```
      f5585834
    • Mixficsol's avatar
      Update cluster.c (#10773) · c751d8a6
      Mixficsol authored
      On line 4068, redis has a logical nodeIsSlave(myself) on the outer if layer,
      which you can delete without having to repeat the decision
      c751d8a6
  3. 03 Jun, 2022 2 commits
  4. 02 Jun, 2022 3 commits
    • zhaozhao.zz's avatar
      rewrite alias config to original name (#10811) · a18c91d6
      zhaozhao.zz authored
      
      
      Redis 7 adds some new alias config like `hash-max-listpack-entries` alias `hash-max-ziplist-entries`.
      
      If a config file contains both real name and alias like this:
      ```
      hash-max-listpack-entries 20
      hash-max-ziplist-entries 20
      ```
      
      after set `hash-max-listpack-entries` to 100 and `config rewrite`, the config file becomes to:
      ```
      hash-max-listpack-entries 100
      hash-max-ziplist-entries 20
      ```
      
      we can see that the alias config is not modified, and users will get wrong config after restart.
      
      6.0 and 6.2 doesn't have this bug, since they only have the `slave` word alias.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      a18c91d6
    • chenyang8094's avatar
      Only print final log when aof is loaded successfully (#10808) · aae0ec25
      chenyang8094 authored
      Skip the print on AOF_NOT_EXIST status.
      aae0ec25
    • zhugezy's avatar
      Fix bugs in CONFIG REWRITE, omitting rename-command and include lines, and... · cf3323db
      zhugezy authored
      
      Fix bugs in CONFIG REWRITE, omitting rename-command and include lines, and inserting comments around module and acl configs (#10761)
      
      A regression from #10285 (redis 7.0).
      CONFIG REWRITE would put lines with: `include`, `rename-command`,
      `user`,  `loadmodule`, and any module specific config in a comment.
      
      For ACL `user`, `loadmodule` and module specific configs would be
      re-inserted at the end (instead of updating existing lines), so the only
      implication is a messy config file full of comments.
      
      But for `rename-command` and `include`, the implication would be that
      they're now missing, so a server restart would lose them.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      cf3323db
  5. 01 Jun, 2022 4 commits
    • Ozan Tezcan's avatar
      Fix Lua compile warning (#10805) · c81b5e55
      Ozan Tezcan authored
      Apparently, GCC 11.2.0 has a new fancy warning for misleading indentations.
      It prints a warning when BRET(b) is on the same line as the loop.
      c81b5e55
    • Oran Agra's avatar
      Expose script flags to processCommand for better handling (#10744) · df558618
      Oran Agra authored
      The important part is that read-only scripts (not just EVAL_RO
      and FCALL_RO, but also ones with `no-writes` executed by normal EVAL or
      FCALL), will now be permitted to run during CLIENT PAUSE WRITE (unlike
      before where only the _RO commands would be processed).
      
      Other than that, some errors like OOM, READONLY, MASTERDOWN are now
      handled by processCommand, rather than the command itself affects the
      error string (and even error code in some cases), and command stats.
      
      Besides that, now the `may-replicate` commands, PFCOUNT and PUBLISH, will
      be considered `write` commands in scripts and will be blocked in all
      read-only scripts just like other write commands.
      They'll also be blocked in EVAL_RO (i.e. even for scripts without the
      `no-writes` shebang flag.
      
      This commit also hides the `may_replicate` flag from the COMMAND command
      output. this is a **breaking change**.
      
      background about may_replicate:
      We don't want to expose a no-may-replicate flag or alike to scripts, since we
      consider the may-replicate thing an internal concern of redis, that we may
      some day get rid of.
      In fact, the may-replicate flag was initially introduced to flag EVAL: since
      we didn't know what it's gonna do ahead of execution, before function-flags
      existed). PUBLISH and PFCOUNT, both of which because they have side effects
      which may some day be fixed differently.
      
      code changes:
      The changes in eval.c are mostly code re-ordering:
      - evalCalcFunctionName is extracted out of evalGenericCommand
      - evalExtractShebangFlags is extracted luaCreateFunction
      - evalGetCommandFlags is new code
      df558618
    • Oran Agra's avatar
      Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check... · b2061de2
      Oran Agra authored
      Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check bug, and new RM_Call checks. (#10786)
      
      * Fix broken protocol when redis can't persist to RDB (general commands, not
        modules), excessive newline. regression of #10372 (7.0 RC3)
      * Fix broken protocol when Redis can't persist to AOF (modules and
        scripts), missing newline.
      * Fix bug in OOM check of EVAL scripts called from RM_Call.
        set the cached OOM state for scripts before executing module commands too,
        so that it can serve scripts that are executed by modules.
        i.e. in the past EVAL executed by RM_Call could have either falsely
        fail or falsely succeeded because of a wrong cached OOM state flag.
      * Fix bugs with RM_Yield:
        1. SHUTDOWN should only accept the NOSAVE mode
        2. Avoid eviction during yield command processing.
        3. Avoid processing master client commands while yielding from another client
      * Add new two more checks to RM_Call script mode.
        1. READONLY You can't write against a read only replica
        2. MASTERDOWN Link with MASTER is down and `replica-serve-stale-data` is set to `no`
      * Add new RM_Call flag to let redis automatically refuse `deny-oom` commands
        while over the memory limit. 
      * Add tests to cover various errors from Scripts, Modules, Modules
        calling scripts, and Modules calling commands in script mode.
      
      Add tests:
      * Looks like the MISCONF error was completely uncovered by the tests,
        add tests for it, including from scripts, and modules
      * Add tests for NOREPLICAS from scripts
      * Add tests for the various errors in module RM_Call, including RM_Call that
        calls EVAL, and RM_call in "eval mode". that includes:
        NOREPLICAS, READONLY, MASTERDOWN, MISCONF
      b2061de2
    • filipe oliveira's avatar
      Moving client flags to a more cache friendly position within client struct (#10697) · 6a6e911f
      filipe oliveira authored
      Move the client flags to a more cache friendly position within the client struct
      we regain the lost 2% of CPU cycles since v6.2 ( from 630532.57 to 647449.80 ops/sec ).
      These are due to higher rate of calls to getClientType due to changes in #9166 and #10020 
      6a6e911f
  6. 31 May, 2022 5 commits
    • menwen's avatar
      Add module API RM_MallocUsableSize (#10795) · 42fbf064
      menwen authored
      This allows the module to know the usable size of an allocation
      it made, rather than the consumed size.
      42fbf064
    • zhaozhao.zz's avatar
      rename channel to shardchannel in sharded pubsub commands (#10738) · 805191c7
      zhaozhao.zz authored
      since the sharded pubsub is different with pubsub, it's better to give users a hint to make it more clear.
      805191c7
    • DarrenJiang13's avatar
      Adds isolated netstats for replication. (#10062) · bb1de082
      DarrenJiang13 authored
      
      
      The amount of `server.stat_net_output_bytes/server.stat_net_input_bytes`
      is actually the sum of replication flow and users' data flow. 
      It may cause confusions like this:
      "Why does my server get such a large output_bytes while I am doing nothing? ". 
      
      After discussions and revisions, now here is the change about what this
      PR brings (final version before merge):
      - 2 server variables to count the network bytes during replication,
           including fullsync and propagate bytes.
           - `server.stat_net_repl_output_bytes`/`server.stat_net_repl_input_bytes`
      - 3 info fields to print the input and output of repl bytes and instantaneous
           value of total repl bytes.
           - `total_net_repl_input_bytes` / `total_net_repl_output_bytes`
           - `instantaneous_repl_total_kbps`
      - 1 new API `rioCheckType()` to check the type of rio. So we can use this
           to distinguish between diskless and diskbased replication
      - 2 new counting items to keep network statistics consistent between master
           and slave
          - rdb portion during diskless replica. in `rdbLoadProgressCallback()`
          - first line of the full sync payload. in `readSyncBulkPayload()`
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      bb1de082
    • Harkrishn Patro's avatar
      Sharded pubsub publish messagebulk as smessage (#10792) · 4065b4f2
      Harkrishn Patro authored
      To easily distinguish between sharded channel message and a global
      channel message, introducing `smessage` (instead of `message`) as
      message bulk for sharded channel publish message.
      
      This is gonna be a breaking change in 7.0.1!
      
      Background:
      Sharded pubsub introduced in redis 7.0, but after the release we quickly
      realized that the fact that it's problematic that the client can't distinguish
      between normal (global) pubsub messages and sharded ones.
      This is important because the same connection can subscribe to both,
      but messages sent to one pubsub system are not propagated to the
      other (they're completely separate), so if one connection is used to
      subscribe to both, we need to assist the client library to know which
      message it got so it can forward it to the correct callback.
      4065b4f2
    • Wen Hui's avatar
      Fix streamParseAddOrTrimArgsOrReply function minor comment issue (#10783) · d7ae8587
      Wen Hui authored
      
      
      When I read the source codes, I have no idea where the option "age" come from.
      Co-authored-by: default avatarUbuntu <lucas.guang.yang1@huawei.com>
      Co-authored-by: default avatarguybe7 <guy.benoish@redislabs.com>
      d7ae8587
  7. 30 May, 2022 1 commit
  8. 29 May, 2022 4 commits
  9. 27 May, 2022 2 commits
    • Vitaly's avatar
      Fix ZRANGESTORE crash when zset_max_listpack_entries is 0 (#10767) · 6461f09f
      Vitaly authored
      When `zrangestore` is called container destination object is created. 
      Before this PR we used to create a listpack based object even if `zset-max-ziplist-entries`
      or equivalent`zset-max-listpack-entries` was set to 0.
      This triggered immediate conversion of the listpack into a skiplist in `zrangestore`, which hits
      an assertion resulting in an engine crash.
      
      Added a TCL test that reproduces this issue.
      6461f09f
    • Binbin's avatar
      Fix some commands key spec in json files (#10779) · 2a099d49
      Binbin authored
      There are some commands that has the wrong key specs.
      This PR adds a key-spec related check in generate-command-code.py.
      Check if the index is valid, or if there is an unused index.
      
      The check result will look like:
      ```
      [root]# python utils/generate-command-code.py
      Processing json files...
      Linking container command to subcommands...
      Checking all commands...
      command: RESTORE_ASKING may have unused key_spec
      command: RENAME may have unused key_spec
      command: PFDEBUG may have unused key_spec
      command: WATCH key_specs missing flags
      command: LCS arg: key2 key_spec_index error
      command: RENAMENX may have unused key_spec
      Error: There are errors in the commands check, please check the above logs.
      ```
      
      The following commands have been fixed according to the check results:
      - RESTORE ASKING: add missing arguments section (and history section)
      - RENAME: newkey's key_spec_index should be 1
      - PFDEBUG: add missing arguments (and change the arity from -3 to 3)
      - WATCH: add missing key_specs flags: RO, like EXIST (it allow you to know the key exists, or is modified, but doesn't "leak" the data)
      - LCS: key2 key_spec_index error, there is only one key-spec
      - RENAMENX: newkey's key_spec_index should be 1
      2a099d49
  10. 26 May, 2022 3 commits
  11. 25 May, 2022 1 commit
  12. 23 May, 2022 1 commit
    • Binbin's avatar
      Fix BZMPOP gets unblocked by non-key args and returns them (#10764) · 450c88f3
      Binbin authored
      This bug was introduced in #9484 (7.0.0).
      It result that BZMPOP blocked on non-key arguments.
      
      Like `bzmpop 0 1 myzset min count 10`, this command will additionally
      block in these keys (except for the first and the last argument) and can return their values:
      - 0: timeout value
      - 1: numkeys value
      - min: min/max token
      - count: count token
      450c88f3
  13. 22 May, 2022 6 commits
    • yoav-steinberg's avatar
      Add warning for suspected slow system clocksource setting (#10636) · 843a4cdc
      yoav-steinberg authored
      This PR does 2 main things:
      1) Add warning for suspected slow system clocksource setting. This is Linux specific.
      2) Add a `--check-system` argument to redis which runs all system checks and prints a report.
      
      ## System checks
      Add a command line option `--check-system` which runs all known system checks and provides
      a report to stdout of which systems checks have failed with details on how to reconfigure the
      system for optimized redis performance.
      The `--system-check` mode exists with an appropriate error code after running all the checks.
      
      ## Slow clocksource details
      We check the system's clocksource performance by running `clock_gettime()` in a loop and then
      checking how much time was spent in a system call (via `getrusage()`). If we spend more than
      10% of the time in the kernel then we print a warning. I verified that using the slow clock sources:
      `acpi_pm` (~90% in the kernel on my laptop) and `xen` (~30% in the kernel on an ec2 `m4.large`)
      we get this warning.
      
      The check runs 5 system ticks so we can detect time spent in kernel at 20% jumps (0%,20%,40%...).
      Anything more accurate will require the test to run longer. Typically 5 ticks are 50ms. This means
      running the test on startup will delay startup by 50ms. To avoid this we make sure the test is only
      executed in the `--check-system` mode.
      
      For a quick startup check, we specifically warn if the we see the system is using the `xen` clocksource
      which we know has bad performance and isn't recommended (at least on ec2). In such a case the
      user should manually rung redis with `--check-system` to force the slower clocksource test described
      above.
      
      ## Other changes in the PR
      
      * All the system checks are now implemented as functions in _syscheck.c_.
        They are implemented using a standard interface (see details in _syscheck.c_).
        To do this I moved the checking functions `linuxOvercommitMemoryValue()`,
        `THPIsEnabled()`, `linuxMadvFreeForkBugCheck()` out of _server.c_ and _latency.c_
        and into the new _syscheck.c_. When moving these functions I made sure they don't
        depend on other functionality provided in _server.c_ and made them use a standard
        "check functions" interface. Specifically:
        * I removed all logging out of `linuxMadvFreeForkBugCheck()`. In case there's some
          unexpected error during the check aborts as before, but without any logging.
          It returns an error code 0 meaning the check didn't not complete.
        * All these functions now return 1 on success, -1 on failure, 0 in case the check itself
          cannot be completed.
        * The `linuxMadvFreeForkBugCheck()` function now internally calls `exit()` and not
          `exitFromChild()` because the latter is only available in _server.c_ and I wanted to
          remove that dependency. This isn't an because we don't need to worry about the
          child process created by the test doing anything related to the rdb/aof files which
          is why `exitFromChild()` was created.
      
      * This also fixes parsing of other /proc/\<pid\>/stat fields to correctly handle spaces
        in the process name and be more robust in general. Not that before this fix the rss
        info in `INFO memory` was corrupt in case of spaces in the process name. To
        recreate just rename `redis-server` to `redis server`, start it, and run `INFO memory`.
      843a4cdc
    • Oran Agra's avatar
      Scripts that declare the `no-writes` flag are implicitly `allow-oom` too. (#10699) · b0e18f80
      Oran Agra authored
      Scripts that have the `no-writes` flag, cannot execute write commands,
      and since all `deny-oom` commands are write commands, we now act
      as if the `allow-oom` flag is implicitly set for scripts that set the `no-writes` flag.
      this also implicitly means that the EVAL*_RO and FCALL_RO commands can
      never fails with OOM error.
      
      Note about a bug that's no longer relevant:
      There was an issue with EVAL*_RO using shebang not being blocked correctly
      in OOM state:
      When an EVAL script declares a shebang, it was by default not allowed to run in
      OOM state.
      but this depends on a flag that is updated before the command is executed, which
      was not updated in case of the `_RO` variants.
      the result is that if the previous cached state was outdated (either true or false),
      the script will either unjustly fail with OOM, or unjustly allowed to run despite
      the OOM state.
      It doesn't affect scripts without a shebang since these depend on the actual
      commands they run, and since these are only read commands, they don't care
      for that cached oom state flag.
      it did affect scripts with shebang and no allow-oom flag, bug after the change in
      this PR, scripts that are run with eval_ro would implicitly have that flag so again
      the cached state doesn't matter.
      
      p.s. this isn't a breaking change since all it does is allow scripts to run when they
      should / could rather than blocking them.
      b0e18f80
    • yoav-steinberg's avatar
      Updated HDR histogram from upsteam after they merged our fix in #10606. (#10755) · cb6933e3
      yoav-steinberg authored
      The code is based on upstream https://github.com/HdrHistogram/HdrHistogram_c
      master branch latest commit (e4448cf6d1cd08fff519812d3b1e58bd5a94ac42).
      The reason to pull this in now is that their final version of our optimization is even faster.
      See: https://github.com/HdrHistogram/HdrHistogram_c/pull/107.
      cb6933e3
    • Binbin's avatar
      Remove ziplist dead code in object.c (#10751) · 18cb4a7d
      Binbin authored
      Remove some dead code in object.c, ziplist is no longer used in 7.0
      
      Some backgrounds:
      zipmap - hash: replaced by ziplist in #285
      ziplist - hash: replaced by listpack in #8887
      ziplist - zset: replaced by listpack in #9366
      ziplist - list: replaced by quicklist (listpack) in #2143 / #9740
      
      Moved the location of ziplist.h in the server.c
      18cb4a7d
    • Yuuoniy's avatar
      Fix memory leak in streamGetEdgeID (#10753) · 4a7a4e42
      Yuuoniy authored
      si is initialized by streamIteratorStart(), we should call
      streamIteratorStop() on it when done.
      
      regression introduced in #9127 (redis 7.0)
      4a7a4e42
    • Ofir Luzon's avatar
      Add SIGINT handler to redis-cli --bigkeys, --memkeys, --hotkeys, --scan (#10736) · 00a9d6b3
      Ofir Luzon authored
      Finish current loop and display the scanned keys summery on SIGINT (Ctrl-C) signal.
      It will also prepend the current scanned percentage to the scanned keys summery 1st line.
      
      In this commit I've renamed and relocated `intrinsicLatencyModeStop` function as I'm using the exact same logic.
      00a9d6b3
  14. 18 May, 2022 1 commit