1. 04 Apr, 2023 1 commit
    • Binbin's avatar
      Changed activeExpireCycle server.masterhost check to iAmMaster in beforeSleep (#11997) · aee8d1ff
      Binbin authored
      In cluster mode, when a node restart as a replica, it doesn't immediately
      sync with the master, replication is enabled in clusterCron. It means that
      sometime server.masterhost is NULL and we wrongly judge it in beforeSleep.
      
      In this case, we may trigger a fast activeExpireCycle in beforeSleep, but the
      node's flag is actually a replica, that can lead to data inconsistency.  In this
      PR, we use iAmMaster to replace the `server.masterhost == NULL`
      
      This is an overlook in #7001, and more discussion in #11783.
      aee8d1ff
  2. 02 Apr, 2023 2 commits
    • Wen Hui's avatar
      redis-cli - handle sensitive command redaction for variadic CONFIG SET (#11975) · a4a0eab5
      Wen Hui authored
      In the Redis 7.0 and newer version,
      config set command support multiply `<parameter> <value>` pairs, thus the previous
      sensitive command condition does not apply anymore
      
      For example:
      
      The command:
      **config set maxmemory 1GB masteruser aa** will be written to redis_cli historyfile
      
      In this PR, we update the condition for these sensitive commands
      config set masteruser <username>
      config set masterauth <master-password>
      config set requirepass foobared
      a4a0eab5
    • Slava Koyfman's avatar
      Disconnect pub-sub subscribers when revoking `allchannels` permission (#11992) · f38aa6bf
      Slava Koyfman authored
      The existing logic for killing pub-sub clients did not handle the `allchannels`
      permission correctly. For example, if you:
      
          ACL SETUSER foo allchannels
      
      Have a client authenticate as the user `foo` and subscribe to a channel, and then:
      
          ACL SETUSER foo resetchannels
      
      The subscribed client would not be disconnected, though new clients under that user
      would be blocked from subscribing to any channels.
      
      This was caused by an incomplete optimization in `ACLKillPubsubClientsIfNeeded`
      checking whether the new channel permissions were a strict superset of the old ones.
      f38aa6bf
  3. 30 Mar, 2023 2 commits
    • Jason Elbaum's avatar
      Reimplement cli hints based on command arg docs (#10515) · 1f76bb17
      Jason Elbaum authored
      
      
      Now that the command argument specs are available at runtime (#9656), this PR addresses
      #8084 by implementing a complete solution for command-line hinting in `redis-cli`.
      
      It correctly handles nearly every case in Redis's complex command argument definitions, including
      `BLOCK` and `ONEOF` arguments, reordering of optional arguments, and repeated arguments
      (even when followed by mandatory arguments). It also validates numerically-typed arguments.
      It may not correctly handle all possible combinations of those, but overall it is quite robust.
      
      Arguments are only matched after the space bar is typed, so partial word matching is not
      supported - that proved to be more confusing than helpful. When the user's current input
      cannot be matched against the argument specs, hinting is disabled.
      
      Partial support has been implemented for legacy (pre-7.0) servers that do not support
      `COMMAND DOCS`, by falling back to a statically-compiled command argument table.
      On startup, if the server does not support `COMMAND DOCS`, `redis-cli` will now issue
      an `INFO SERVER` command to retrieve the server version (unless `HELLO` has already
      been sent, in which case the server version will be extracted from the reply to `HELLO`).
      The server version will be used to filter the commands and arguments in the command table,
      removing those not supported by that version of the server. However, the static table only
      includes core Redis commands, so with a legacy server hinting will not be supported for
      module commands. The auto generated help.h and the scripts that generates it are gone.
      
      Command and argument tables for the server and CLI use different structs, due primarily
      to the need to support different runtime data. In order to generate code for both, macros
      have been added to `commands.def` (previously `commands.c`) to make it possible to
      configure the code generation differently for different use cases (one linked with redis-server,
      and one with redis-cli).
      
      Also adding a basic testing framework for the command hints based on new (undocumented)
      command line options to `redis-cli`: `--test_hint 'INPUT'` prints out the command-line hint for
      a given input string, and `--test_hint_file <filename>` runs a suite of test cases for the hinting
      mechanism. The test suite is in `tests/assets/test_cli_hint_suite.txt`, and it is run from
      `tests/integration/redis-cli.tcl`.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      1f76bb17
    • Madelyn Olson's avatar
      Fixed tracking of command duration for multi/eval/module/wait (#11970) · 971b177f
      Madelyn Olson authored
      In #11012, we changed the way command durations were computed to handle the same command being executed multiple times. This commit fixes some misses from that commit.
      
      * Wait commands were not correctly reporting their duration if the timeout was reached.
      * Multi/scripts/and modules with RM_Call were not properly resetting the duration between inner calls, leading to them reporting cumulative duration.
      * When a blocked client is freed, the call and duration are always discarded.
      
      This commit also adds an assert if the duration is not properly reset, potentially indicating that a report to call statistics was missed. The assert potentially be removed in the future, as it's mainly intended to detect misses in tests.
      971b177f
  4. 29 Mar, 2023 2 commits
    • Itamar Haber's avatar
      Overhauls command summaries and man pages. (#11942) · 0c3b8b7e
      Itamar Haber authored
      This is an attempt to normalize/formalize command summaries.
      
      Main actions performed:
      
      * Starts with the continuation of the phrase "The XXXX command, when called, ..." for user commands.
      * Starts with "An internal command...", "A container command...", etc... when applicable.
      * Always uses periods.
      * Refrains from referring to other commands. If this is needed, backquotes should be used for command names.
      * Tries to be very clear about the data type when applicable.
      * Tries to mention additional effects, e.g. "The key is created if it doesn't exist" and "The set is deleted if the last member is removed."
      * Prefers being terse over verbose.
      * Tries to be consistent.
      0c3b8b7e
    • Binbin's avatar
      Fix fork done handler wrongly update fsync metrics and enhance AOF_ FSYNC_ALWAYS (#11973) · cb171786
      Binbin authored
      This PR fix several unrelated bugs that were discovered by the same set of tests
      (WAITAOF tests in #11713), could make the `WAITAOF` test hang. 
      
      The change in `backgroundRewriteDoneHandler` is about MP-AOF.
      That leftover / old code assumes that we started a new AOF file just now
      (when we have a new base into which we're gonna incrementally write), but
      the fact is that with MP-AOF, the fork done handler doesn't really affect the
      incremental file being maintained by the parent process, there's no reason to
      re-issue `SELECT`, and no reason to update any of the fsync variables in that flow.
      This should have been deleted with MP-AOF (introduced in #9788, 7.0).
      The damage is that the update to `aof_fsync_offset` will cause us to miss an fsync
      in `flushAppendOnlyFile`, that happens if we stop write commands in `AOF_FSYNC_EVERYSEC`
      while an AOFRW is in progress. This caused a new `WAITAOF` test to sometime hang forever.
      
      Also because of MP-AOF, we needed to change `aof_fsync_offset` to `aof_last_incr_fsync_offset`
      and match it to `aof_last_incr_size` in `flushAppendOnlyFile`. This is because in the past we compared
      `aof_fsync_offset` and `aof_current_size`, but with MP-AOF it could be the total AOF file will be
      smaller after AOFRW, and the (already existing) incr file still has data that needs to be fsynced.
      
      The change in `flushAppendOnlyFile`, about the `AOF_FSYNC_ALWAYS`, it is follow #6053
      (the details is in #5985), we also check `AOF_FSYNC_ALWAYS` to handle a case where
      appendfsync is changed from everysec to always while there is data that's written but not yet fsynced.
      cb171786
  5. 27 Mar, 2023 1 commit
    • Rafi Einstein's avatar
      Clang: fix for -flto argument (#11961) · 557ca05d
      Rafi Einstein authored
      Starting with the recent #11926 Makefile specifies `-flto=auto` which is unsupported on clang.
      Additionally, detecting clang correctly requires actually running it, since on MacOS gcc can be an alias for clang.
      557ca05d
  6. 26 Mar, 2023 2 commits
    • Binbin's avatar
      Fix redis-cli cluster test timing issue (#11887) · aa2403ca
      Binbin authored
      This test fails sporadically:
      ```
      *** [err]: Migrate the last slot away from a node using redis-cli in tests/unit/cluster/cli.tcl
      cluster size did not reach a consistent size 4
      ```
      
      I guess the time (5s) of wait_for_cluster_size is not enough,
      usually, the waiting time for our other tests for cluster
      consistency is 50s, so also changing it to 50s.
      aa2403ca
    • Binbin's avatar
      Add COMMAND COUNT test to cover reply-schemas-validator test (#11971) · 2cc99c69
      Binbin authored
      Since we remove the COMMAND COUNT call in sentinel test in #11950,
      reply-schemas-validator started reporting this error:
      ```
      WARNING! The following commands were not hit at all:
        command|count
        ERROR! at least one command was not hit by the tests
      ```
      
      This PR add a COMMAND COUNT test to cover it and also fix some
      typos in req-res-log-validator.py
      2cc99c69
  7. 23 Mar, 2023 2 commits
  8. 22 Mar, 2023 6 commits
  9. 21 Mar, 2023 3 commits
    • Roshan Khatri's avatar
      Module commands to have ACL categories. (#11708) · 6948daca
      Roshan Khatri authored
      
      
      This allows modules to register commands to existing ACL categories and blocks the creation of [sub]commands, datatypes and registering the configs outside of the OnLoad function.
      
      For allowing modules to register commands to existing ACL categories,
      This PR implements a new API int RM_SetCommandACLCategories() which takes a pointer to a RedisModuleCommand and a C string aclflags containing the set of space separated ACL categories.
      Example, 'write slow' marks the command as part of the write and slow ACL categories.
      
      The C string aclflags is tokenized by implementing a helper function categoryFlagsFromString(). Theses tokens are matched and the corresponding ACL categories flags are set by a helper function matchAclCategoriesFlags. The helper function categoryFlagsFromString() returns the corresponding categories_flags or returns -1 if some token not processed correctly.
      
      If the module contains commands which are registered to existing ACL categories, the number of [sub]commands are tracked by num_commands_with_acl_categories in struct RedisModule. Further, the allowed command bit-map of the existing users are recomputed from the command_rules list, by implementing a function called ACLRecomputeCommandBitsFromCommandRulesAllUsers() for the existing users to have access to the module commands on runtime.
      
      ## Breaking change
      This change requires that registering commands and subcommands only occur during a modules "OnLoad" function, in order to allow efficient recompilation of ACL bits. We also chose to block registering configs and types, since we believe it's only valid for those to be created during onLoad. We check for this onload flag in struct RedisModule to check if the call is made from the OnLoad function.
      Co-authored-by: default avatarMadelyn Olson <madelyneolson@gmail.com>
      6948daca
    • Binbin's avatar
      Fix race in temp rdb delete shutdown test (#11840) · 78f15b7e
      Binbin authored
      I saw this error once, in the FreeBSD Daily CI:
      ```
      *** [err]: Temp rdb will be deleted if we use bg_unlink when shutdown in tests/unit/shutdown.tcl
      Expected [file exists /xxx/temp-10336.rdb] (context: type eval line 15 cmd {assert {[file exists $temp_rdb]}} proc ::test)
      ```
      
      The log shows that bgsave was executed, and it was successfully executed in the end:
      ```
      Starting test Temp rdb will be deleted if we use bg_unlink when shutdown in tests/unit/shutdown.tcl
      10251:M 22 Feb 2023 11:37:25.441 * Background saving started by pid 10336
      10336:C 22 Feb 2023 11:37:27.949 * DB saved on disk
      10336:C 22 Feb 2023 11:37:27.949 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
      10251:M 22 Feb 2023 11:37:28.060 * Background saving terminated with success
      ```
      
      There may be two reasons:
      1. The child process has been created, but it has not created
         the temp rdb file yet, so [file exists $temp_rdb] check failed.
      2. The child process bgsave has been executed successfully and the
         temp file has been deleted, so [file exists $temp_rdb] check failed.
      
      From the logs pint, it should be the case 2, case 1 is too extreme,
      set rdb-key-save-delay to a higher value to ensure bgsave does not
      succeed early to avoid this case.
      78f15b7e
    • Binbin's avatar
      Add missing master_reboot flag in sentinel instance info (#11888) · 1cb4b1ad
      Binbin authored
      SRI_MASTER_REBOOT flag was added in #9438
      1cb4b1ad
  10. 20 Mar, 2023 4 commits
    • Oran Agra's avatar
      Avoid assertion when MSETNX is used with the same key twice (CVE-2023-28425) (#11940) · 48e0d478
      Oran Agra authored
      Using the same key twice in MSETNX command would trigger an assertion.
      
      This reverts #11594 (introduced in Redis 7.0.8)
      48e0d478
    • Binbin's avatar
      Fix new subscribe mode test in reply-schemas-validator (#11939) · c9124145
      Binbin authored
      The reason is in reply-schemas-validator, the resp of the
      client we create will be client_default_resp (currently 3):
      ```
      client *createClient(connection *conn) {
          client *c = zmalloc(sizeof(client));
       #ifdef LOG_REQ_RES
          reqresReset(c, 0);
          c->resp = server.client_default_resp;
       #else
          c->resp = 2;
       #endif
      }
      ```
      
      But current_resp3 in redis-cli will be inconsistent with it,
      the test adds a simple hello 3 to avoid this failure, test
      was added in #11873.
      
      Added help descriptions for dont-pre-clean option, it was
      added in #10273
      c9124145
    • polaris-alioth's avatar
      passwords printed in the crash log (#11930) · 56eef6fb
      polaris-alioth authored
      When the server crashes during the AUTH command, or another command with
      an AUTH argument, the password was recorded in the log.
      
      Now, when the `auth` keyword is detected (could be in HELLO or MIGRATE, etc),
      the loop exits before printing any additional arguments.
      56eef6fb
    • Shaya Potter's avatar
      Don't run command filter on blocked command reprocessing (#11895) · 6cf8fc08
      Shaya Potter authored
      
      
      Previously we would run the module command filters even upon blocked
      command reprocessing.  This could modify the command, and it's args.
      This is irrelevant in the context of a command being reprocessed (it already
      went through the filters), as well as breaks the crashed command lookup
      that exists in the case of a reprocessed command.
      
      fixes #11894.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      6cf8fc08
  11. 19 Mar, 2023 2 commits
    • Viktor Söderqvist's avatar
      redis-cli: Accept commands in subscribed mode (#11873) · bbf364a4
      Viktor Söderqvist authored
      The message "Reading messages... (press Ctrl-C to quit)" is replaced by
      "Reading messages... (press Ctrl-C to quit or any key to type command)".
      
      This allows users to subscribe to more channels, to try out UNSUBSCRIBE and to
      combine pubsub with other features such as push messages from client tracking.
      
      The "Reading messages" info message is displayed in the bottom of the output in a
      distinct style and moves downward as more messages appear. When any key is pressed,
      the info message is replaced by the prompt with for entering commands.
      After entering a command and the reply is displayed, the "Reading messages" info
      messages appears again. This is added to the repl loop in redis-cli and in the
      corresponding place for non-interactive mode.
      
      An indication "(subscribed mode)" is included in the prompt when entering commands
      in subscribed mode.
      
      Also:
      * Fixes a problem that UNSUBSCRIBE hanged when used with RESP3 and push callback,
        without first entering subscribe mode. It hanged because UNSUBSCRIBE gets one or
        more push replies but no in-band reply.
      * Exit subscribed mode after RESET.
      bbf364a4
    • Wang Yuan's avatar
      Remove unnecessary `fsync` when sentinel flushs config file (#11910) · c9466b24
      Wang Yuan authored
      `rewriteConfig` already calls `fsync` to make sure changes are committed to disk.
      so it is no need to call `fsync` again here.
      this was added here when rewriteConfigOverwriteFile used the ftruncate approach and didn't fsync
      c9466b24
  12. 17 Mar, 2023 2 commits
    • Rong Tao's avatar
      Fix compile lto-wrapper warning for aarch64 (#11926) · d6910983
      Rong Tao authored
      
      
      Use -flto=auto to use GNU make's job server, if available, or otherwise fall
      back to autodetection of the number of CPU threads present in your system.
      
        Warnings:
      
        lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
        lto-wrapper: note: see the ‘-flto’ option documentation for more information
        lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
        lto-wrapper: note: see the ‘-flto’ option documentation for more information
        lto-wrapper: warning: using serial compilation of 31 LTRANS jobs
        lto-wrapper: note: see the ‘-flto’ option documentation for more information
      Signed-off-by: default avatarRong Tao <rongtao@cestc.cn>
      d6910983
    • Binbin's avatar
      Minor fix to print, set to str (#11934) · 2dd5c3a1
      Binbin authored
      * Minor fix to print, set to str
      
      `{commands_filename}` the extra {} actually make it
      become a Set, and the output print was like this:
      ```
      Processing json files...
      Linking container command to subcommands...
      Checking all commands...
      Generating {'commands'}.c...
      All done, exiting.
      ```
      
      Introduced in #11920
      
      * more fix
      2dd5c3a1
  13. 16 Mar, 2023 3 commits
    • Meir Shpilraien (Spielrein)'s avatar
      Support for RM_Call on blocking commands (#11568) · d0da0a6a
      Meir Shpilraien (Spielrein) authored
      Allow running blocking commands from within a module using `RM_Call`.
      
      Today, when `RM_Call` is used, the fake client that is used to run command
      is marked with `CLIENT_DENY_BLOCKING` flag. This flag tells the command
      that it is not allowed to block the client and in case it needs to block, it must
      fallback to some alternative (either return error or perform some default behavior).
      For example, `BLPOP` fallback to simple `LPOP` if it is not allowed to block.
      
      All the commands must respect the `CLIENT_DENY_BLOCKING` flag (including
      module commands). When the command invocation finished, Redis asserts that
      the client was not blocked.
      
      This PR introduces the ability to call blocking command using `RM_Call` by
      passing a callback that will be called when the client will get unblocked.
      In order to do that, the user must explicitly say that he allow to perform blocking
      command by passing a new format specifier argument, `K`, to the `RM_Call`
      funct...
      d0da0a6a
    • Binbin's avatar
      Fix usleep compilation warning in auth.c (#11925) · 484b73a8
      Binbin authored
      There is a -Wimplicit-function-declaration warning in here:
      ```
      auth.c: In function ‘AuthBlock_ThreadMain’:
      auth.c:116:5: warning: implicit declaration of function ‘usleep’; did you mean ‘sleep’? [-Wimplicit-function-declaration]
        116 |     usleep(500000);
            |     ^~~~~~
            |     sleep
      ```
      484b73a8
    • Binbin's avatar
      Bump codespell to 2.2.4, fix typos and outupdated comments (#11911) · 0b159b34
      Binbin authored
      Fix some seen typos and wrong comments.
      0b159b34
  14. 15 Mar, 2023 5 commits
    • KarthikSubbarao's avatar
      Custom authentication for Modules (#11659) · f8a5a4f7
      KarthikSubbarao authored
      
      
      This change adds new module callbacks that can override the default password based authentication associated with ACLs. With this, Modules can register auth callbacks through which they can implement their own Authentication logic. When `AUTH` and `HELLO AUTH ...` commands are used, Module based authentication is attempted and then normal password based authentication is attempted if needed.
      The new Module APIs added in this PR are - `RM_RegisterCustomAuthCallback` and `RM_BlockClientOnAuth` and `RedisModule_ACLAddLogEntryByUserName `.
      
      Module based authentication will be attempted for all Redis users (created through the ACL SETUSER cmd or through Module APIs) even if the Redis user does not exist at the time of the command. This gives a chance for the Module to create the RedisModule user and then authenticate via the RedisModule API - from the custom auth callback.
      
      For the AUTH command, we will support both variations - `AUTH <username> <password>` and `AUTH <password>`. In case of the `AUTH <password>` variation, the custom auth callbacks are triggered with “default” as the username and password as what is provided.
      
      
      ### RedisModule_RegisterCustomAuthCallback
      ```
      void RM_RegisterCustomAuthCallback(RedisModuleCtx *ctx, RedisModuleCustomAuthCallback cb) {
      ```
      This API registers a callback to execute to prior to normal password based authentication. Multiple callbacks can be registered across different modules. These callbacks are responsible for either handling the authentication, each authenticating the user or explicitly denying, or deferring it to other authentication mechanisms. Callbacks are triggered in the order they were registered. When a Module is unloaded, all the auth callbacks registered by it are unregistered. The callbacks are attempted, in the order of most recently registered callbacks, when the AUTH/HELLO (with AUTH field is provided) commands are called. The callbacks will be called with a module context along with a username and a password, and are expected to take one of the following actions:
      
       (1) Authenticate - Use the RM_Authenticate* API successfully and return `REDISMODULE_AUTH_HANDLED`. This will immediately end the auth chain as successful and add the OK reply.
      (2) Block a client on authentication - Use the `RM_BlockClientOnAuth` API and return `REDISMODULE_AUTH_HANDLED`. Here, the client will be blocked until the `RM_UnblockClient `API is used which will trigger the auth reply callback (provided earlier through the `RM_BlockClientOnAuth`). In this reply callback, the Module should authenticate, deny or skip handling authentication.
      (3) Deny Authentication - Return `REDISMODULE_AUTH_HANDLED` without authenticating or blocking the client. Optionally, `err` can be set to a custom error message. This will immediately end the auth chain as unsuccessful and add the ERR reply.
      (4) Skip handling Authentication - Return `REDISMODULE_AUTH_NOT_HANDLED` without blocking the client. This will allow the engine to attempt the next custom auth callback.
      
      If none of the callbacks authenticate or deny auth, then password based auth is attempted and will authenticate or add failure logs and reply to the clients accordingly.
      
      ### RedisModule_BlockClientOnAuth
      ```
      RedisModuleBlockedClient *RM_BlockClientOnAuth(RedisModuleCtx *ctx, RedisModuleCustomAuthCallback reply_callback,
                                                     void (*free_privdata)(RedisModuleCtx*,void*))
      ```
      This API can only be used from a Module from the custom auth callback. If a client is not in the middle of custom module based authentication, ERROR is returned. Otherwise, the client is blocked and the `RedisModule_BlockedClient` is returned similar to the `RedisModule_BlockClient` API.
      
      ### RedisModule_ACLAddLogEntryByUserName
      ```
      int RM_ACLAddLogEntryByUserName(RedisModuleCtx *ctx, RedisModuleString *username, RedisModuleString *object, RedisModuleACLLogEntryReason reason)
      ```
      Adds a new entry in the ACL log with the `username` RedisModuleString provided. This simplifies the Module usage because now, developers do not need to create a Module User just to add an error ACL Log entry. Aside from accepting username (RedisModuleString) instead of a RedisModuleUser, it is the same as the existing `RedisModule_ACLAddLogEntry` API.
      
      
      ### Breaking changes
      - HELLO command - Clients can now only set the client name and RESP protocol from the `HELLO` command if they are authenticated. Also, we now finish command arg validation first and return early with a ERR reply if any arg is invalid. This is to avoid mutating the client name / RESP from a command that would have failed on invalid arguments.
      
      ### Notable behaviors
      - Module unblocking - Now, we will not allow Modules to block the client from inside the context of a reply callback (triggered from the Module unblock flow `moduleHandleBlockedClients`).
      
      ---------
      Co-authored-by: default avatarMadelyn Olson <34459052+madolson@users.noreply.github.com>
      f8a5a4f7
    • Binbin's avatar
      Fix WAITAOF mix-use last_offset and last_numreplicas (#11922) · 58285a6e
      Binbin authored
      There be a situation that satisfies WAIT, and then wrongly unblock
      WAITAOF because we mix-use last_offset and last_numreplicas.
      
      We update last_offset and last_numreplicas only when the condition
      matches. i.e. output of either replicationCountAOFAcksByOffset or
      replicationCountAcksByOffset is right.
      
      In this case, we need to have separate last_ variables for each of
      them. Added a last_aof_offset and last_aof_numreplicas for WAITAOF.
      
      WAITAOF was added in #11713. Found while coding #11917.
      A Test was added to validate that case.
      58285a6e
    • Ozan Tezcan's avatar
      Use older string format to support earlier python versions (#11920) · 72f5aad0
      Ozan Tezcan authored
      Redis build runs `utils/generate-command-code.py` if there is a change in `src/commands/*.json` files. 
      
      In https://github.com/redis/redis/pull/10273, we used f-string format in this script. f-string feature was introduced in python3.6. 
      If a system has an earlier python version, build might fail. 
      
      Added some changes to make that script compatible with earlier python versions. 
      72f5aad0
    • Binbin's avatar
      Fix WAITAOF reply when using last_offset and last_numreplicas (#11917) · 70b2c4f5
      Binbin authored
      WAITAOF wad added in #11713, its return is an array.
      But forget to handle WAITAOF in last_offset and last_numreplicas,
      causing WAITAOF to return a WAIT like reply.
      
      Tests was added to validate that case (both WAIT and WAITAOF).
      This PR also refactored processClientsWaitingReplicas a bit for better
      maintainability and readability.
      70b2c4f5
    • Kaige Ye's avatar
      cleanup NBSP characters in comments (#10555) · 5360350e
      Kaige Ye authored
      Replace NBSP character (0xC2 0xA0) with space (0x20).
      
      Looks like that was originally added due to misconfigured editor which seems to have been fixed by now.
      5360350e
  15. 14 Mar, 2023 1 commit
    • Slava Koyfman's avatar
      Implementing the WAITAOF command (issue #10505) (#11713) · 9344f654
      Slava Koyfman authored
      Implementing the WAITAOF functionality which would allow the user to
      block until a specified number of Redises have fsynced all previous write
      commands to the AOF.
      
      Syntax: `WAITAOF <num_local> <num_replicas> <timeout>`
      Response: Array containing two elements: num_local, num_replicas
      num_local is always either 0 or 1 representing the local AOF on the master.
      num_replicas is the number of replicas that acknowledged the a replication
      offset of the last write being fsynced to the AOF.
      
      Returns an error when called on replicas, or when called with non-zero
      num_local on a master with AOF disabled, in all other cases the response
      just contains number of fsync copies.
      
      Main changes:
      * Added code to keep track of replication offsets that are confirmed to have
        been fsynced to disk.
      * Keep advancing master_repl_offset even when replication is disabled (and
        there's no replication backlog, only if there's an AOF enabled).
        This way we ca...
      9344f654
  16. 13 Mar, 2023 1 commit
    • Binbin's avatar
      Fix tail->repl_offset update in feedReplicationBuffer (#11905) · 7997874f
      Binbin authored
      
      
      In #11666, we added a while loop and will split a big reply
      node to multiple nodes. The update of tail->repl_offset may
      be wrong. Like before #11666, we would have created at most
      one new reply node, and now we will create multiple nodes if
      it is a big reply node.
      
      Now we are creating more than one node, and the tail->repl_offset
      of all the nodes except the last one are incorrect. Because we
      update master_repl_offset at the beginning, and then use it to
      update the tail->repl_offset. This would have lead to an assertion
      during PSYNC, a test was added to validate that case.
      
      Besides that, the calculation of size was adjusted to fix
      tests that failed due to a combination of a very low backlog size,
      and some thresholds of that get violated because of the relatively
      high overhead of replBufBlock. So now if the backlog size / 16 is too
      small, we'll take PROTO_REPLY_CHUNK_BYTES instead.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      7997874f
  17. 12 Mar, 2023 1 commit
    • xbasel's avatar
      Large blocks of replica client output buffer could lead to psync loops and... · 7be7834e
      xbasel authored
      
      Large blocks of replica client output buffer could lead to psync loops and unnecessary memory usage (#11666)
      
      This can happen when a key almost equal or larger than the
      client output buffer limit of the replica is written.
      
      Example:
      1. DB is empty
      2. Backlog size is 1 MB
      3. Client out put buffer limit is 2 MB
      4. Client writes a 3 MB key
      5. The shared replication buffer will have a single node which contains
      the key written above, and it exceeds the backlog size.
      
      At this point the client output buffer usage calculation will report the
      replica buffer to be 3 MB (or more) even after sending all the data to
      the replica.
      The primary drops the replica connection for exceeding the limits,
      the replica reconnects and successfully executes partial sync but the
      primary will drop the connection again because the buffer usage is still
      3 MB. This happens over and over.
      
      To mitigate the problem, this fix limits the maximum size of a single
      backlog node to be (repl_backlog_size/16). This way a single node can't
      exceed the limits of the COB (the COB has to be larger than the
      backlog).
      It also means that if the backlog has some excessive data it can't trim,
      it would be at most about 6% overuse.
      
      other notes:
      1. a loop was added in feedReplicationBuffer which caused a massive LOC
        change due to indentation, the actual changes are just the `min(max` and the loop.
      3. an unrelated change in an existing test to speed up a server termination which took 10 seconds.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      7be7834e