1. 11 May, 2022 1 commit
    • Binbin's avatar
      FLUSHDB and FLUSHALL add call forceCommandPropagation / FLUSHALL reset dirty... · 783b210d
      Binbin authored
      FLUSHDB and FLUSHALL add call forceCommandPropagation / FLUSHALL reset dirty counter to 0 if we enable save (#10691)
      
      ## FLUSHALL
      We used to restore the dirty counter after `rdbSave` zeroed it if we enable save.
      Otherwise FLUSHALL will not be replicated nor put into the AOF.
      
      And then we do increment it again below.
      Without that extra dirty++, when db was already empty, FLUSHALL
      will not be replicated nor put into the AOF.
      
      We now gonna replace all that dirty counter magic with a call
      to forceCommandPropagation (REPL and AOF), instead of all the
      messing around with the dirty counter.
      Added tests to cover three part (dirty counter, REPL, AOF).
      
      One benefit other than cleaner code is that the `rdb_changes_since_last_save` is correct in this case.
      
      ## FLUSHDB
      FLUSHDB was not replicated nor put into the AOF when db was already empty.
      Unlike DEL on a non-existing key, FLUSHDB always does something, and that's to call the module hook. 
      So basically FLUSHDB is never a NOP, and thus it should always be propagated.
      Not doing that, could mean that if a module does something in that hook, and wants to
      avoid issues of that hook being missing on the replica if the db is empty, it'll need to do complicated things.
      
      So now FLUSHDB add call forceCommandPropagation, we will always propagate FLUSHDB.
      Always propagating FLUSHDB seems like a safe approach that shouldn't have any drawbacks (other than looking odd)
      
      This was mentioned in #8972
      
      ## Test section:
      We actually found it while solving a race condition in the BGSAVE test (other.tcl).
      It was found in extra_ci Daily Arm64 (test-libc-malloc).
      ```
      [exception]: Executing test client: ERR Background save already in progress.
      ERR Background save already in progress
      ```
      
      It look like `r flushdb` trigger (schedule) a bgsave right after `waitForBgsave r` and before `r save`.
      Changing flushdb to flushall, FLUSHALL will do a foreground save and then set the dirty counter to 0.
      783b210d
  2. 17 Feb, 2022 1 commit
    • chenyang8094's avatar
      Adapt redis-check-aof tool for Multi Part Aof (#10061) · a50aa29b
      chenyang8094 authored
      Modifications of this PR:
      1. Support the verification of `Multi Part AOF`, while still maintaining support for the
        old-style `AOF/RDB-preamble`. `redis-check-aof` will automatically choose which
        mode to use according to the incoming file format.
         
      `Usage: redis-check-aof [--fix|--truncate-to-timestamp $timestamp] <AOF/manifest>`
       
      2. Refactor part of the code to make it easier to understand
      3. Currently only supports truncate  (`--fix` or `--truncate-to-timestamp`) the last AOF
        file (may be `BASE` or `INCR`)
      
      The reasons for 3 above:
      - for `--fix`: Only the last AOF may be truncated, this is guaranteed by redis
      - for `--truncate-to-timestamp`:  Normally, we only have `BASE` + `INCR` files
        at most, and `BASE` cannot be truncated(It only contains a timestamp annotation
        at the beginning of the file), so only `INCR` can be truncated. If we have a
        `BASE+INCR1+INCR2` file (meaning we have an interrupted AOFRW), Only `INCR2`
        files can be truncated at this time. If we still insist on truncate `INCR1`, we need to
        manually delete `INCR2` and update the manifest file, then re-run `redis-check-aof`
      - If we want to support truncate any file, we need to add very complicated code to support
        the atomic modification of multiple file deletion and update manifest, I think this is unnecessary
      a50aa29b
  3. 24 Jan, 2022 2 commits
  4. 03 Jan, 2022 1 commit
    • chenyang8094's avatar
      Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788) · 87789fae
      chenyang8094 authored
      
      
      Implement Multi-Part AOF mechanism to avoid overheads during AOFRW.
      Introducing a folder with multiple AOF files tracked by a manifest file.
      
      The main issues with the the original AOFRW mechanism are:
      * buffering of commands that are processed during rewrite (consuming a lot of RAM)
      * freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.
      * double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)
      
      The main modifications of this PR:
      1. Remove the AOF rewrite buffer and related code.
      2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type,
        it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only
        one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the
        incremental commands since the last AOFRW.
      3. Use a AOF manifest file to record and manage these AOF files mentioned above.
      4. The original configuration of `appendfilename` will be the base part of the new file name, for example:
        `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof`
      5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename`
      6. Remove the `aof_rewrite_buffer_length` field in info.
      7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs.
        It also gives users the opportunity to preserve the history AOFs. just for testing use now.
      8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now),
        we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be
        delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit
        period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately.
      9. Support upgrade (load) data from old version redis.
      10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and
        manifest file will be placed in this directory.
      11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if
        `aof-load-truncated` is enabled.
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      87789fae
  5. 21 Dec, 2021 1 commit
    • zhugezy's avatar
      Remove EVAL script verbatim replication, propagation, and deterministic execution logic (#9812) · 1b0968df
      zhugezy authored
      
      
      # Background
      
      The main goal of this PR is to remove relevant logics on Lua script verbatim replication,
      only keeping effects replication logic, which has been set as default since Redis 5.0.
      As a result, Lua in Redis 7.0 would be acting the same as Redis 6.0 with default
      configuration from users' point of view.
      
      There are lots of reasons to remove verbatim replication.
      Antirez has listed some of the benefits in Issue #5292:
      
      >1. No longer need to explain to users side effects into scripts.
          They can do whatever they want.
      >2. No need for a cache about scripts that we sent or not to the slaves.
      >3. No need to sort the output of certain commands inside scripts
          (SMEMBERS and others): this both simplifies and gains speed.
      >4. No need to store scripts inside the RDB file in order to startup correctly.
      >5. No problems about evicting keys during the script execution.
      
      When looking back at Redis 5.0, antirez and core team decided to set the config
      `lua-replicate-commands yes` by default instead of removing verbatim replication
      directly, in case some bad situations happened. 3 years later now before Redis 7.0,
      it's time to remove it formally.
      
      # Changes
      
      - configuration for lua-replicate-commands removed
        - created config file stub for backward compatibility
      - Replication script cache removed
        - this is useless under script effects replication
        - relevant statistics also removed
      - script persistence in RDB files is also removed
      - Propagation of SCRIPT LOAD and SCRIPT FLUSH to replica / AOF removed
      - Deterministic execution logic in scripts removed (i.e. don't run write commands
        after random ones, and sorting output of commands with random order)
        - the flags indicating which commands have non-deterministic results are kept as hints to clients.
      - `redis.replicate_commands()` & `redis.set_repl()` changed
        - now `redis.replicate_commands()` does nothing and return an 1
        - ...and then `redis.set_repl()` can be issued before `redis.replicate_commands()` now
      - Relevant TCL cases adjusted
      - DEBUG lua-always-replicate-commands removed
      
      # Other changes
      - Fix a recent bug comparing CLIENT_ID_AOF to original_client->flags instead of id. (introduced in #9780)
      Co-authored-by: default avatarOran Agra <oran@redislabs.com>
      1b0968df
  6. 26 Oct, 2021 1 commit
  7. 25 Oct, 2021 1 commit
    • Wang Yuan's avatar
      Add timestamp annotations in AOF (#9326) · 9ec3294b
      Wang Yuan authored
      Add timestamp annotation in AOF, one part of #9325.
      
      Enabled with the new `aof-timestamp-enabled` config option.
      
      Timestamp annotation format is "#TS:${timestamp}\r\n"."
      TS" is short of timestamp and this method could save extra bytes in AOF.
      
      We can use timestamp annotation for some special functions. 
      - know the executing time of commands
      - restore data to a specific point-in-time (by using redis-check-rdb to truncate the file)
      9ec3294b
  8. 23 Sep, 2021 1 commit
    • Binbin's avatar
      Add ZMPOP/BZMPOP commands. (#9484) · 14d6abd8
      Binbin authored
      This is similar to the recent addition of LMPOP/BLMPOP (#9373), but zset.
      
      Syntax for the new ZMPOP command:
      `ZMPOP numkeys [<key> ...] MIN|MAX [COUNT count]`
      
      Syntax for the new BZMPOP command:
      `BZMPOP timeout numkeys [<key> ...] MIN|MAX [COUNT count]`
      
      Some background:
      - ZPOPMIN/ZPOPMAX take only one key, and can return multiple elements.
      - BZPOPMIN/BZPOPMAX take multiple keys, but return only one element from just one key.
      - ZMPOP/BZMPOP can take multiple keys, and can return multiple elements from just one key.
      
      Note that ZMPOP/BZMPOP can take multiple keys, it eventually operates on just on key.
      And it will propagate as ZPOPMIN or ZPOPMAX with the COUNT option.
      
      As new commands, if we can not pop any elements, the response like:
      - ZMPOP: Return a NIL in both RESP2 and RESP3, unlike ZPOPMIN/ZPOPMAX return emptyarray.
      - BZMPOP: Return a NIL in both RESP2 and RESP3 when timeout is reached, like BZPOPMIN/BZPOPMAX.
      
      For the normal response is nested arrays in RESP2 and RESP3:
      ```
      ZMPOP/BZMPOP
      1) keyname
      2) 1) 1) member1
            2) score1
         2) 1) member2
            2) score2
      
      In RESP2:
      1) "myzset"
      2) 1) 1) "three"
            2) "3"
         2) 1) "two"
            2) "2"
      
      In RESP3:
      1) "myzset"
      2) 1) 1) "three"
            2) (double) 3
         2) 1) "two"
            2) (double) 2
      ```
      14d6abd8
  9. 09 Sep, 2021 1 commit
    • Binbin's avatar
      Add LMPOP/BLMPOP commands. (#9373) · c50af0ae
      Binbin authored
      We want to add COUNT option for BLPOP.
      But we can't do it without breaking compatibility due to the command arguments syntax.
      So this commit introduce two new commands.
      
      Syntax for the new LMPOP command:
      `LMPOP numkeys [<key> ...] LEFT|RIGHT [COUNT count]`
      
      Syntax for the new BLMPOP command:
      `BLMPOP timeout numkeys [<key> ...] LEFT|RIGHT [COUNT count]`
      
      Some background:
      - LPOP takes one key, and can return multiple elements.
      - BLPOP takes multiple keys, but returns one element from just one key.
      - LMPOP can take multiple keys and return multiple elements from just one key.
      
      Note that LMPOP/BLMPOP  can take multiple keys, it eventually operates on just one key.
      And it will propagate as LPOP or RPOP with the COUNT option.
      
      As a new command, it still return NIL if we can't pop any elements.
      For the normal response is nested arrays in RESP2 and RESP3, like:
      ```
      LMPOP/BLMPOP 
      1) keyname
      2) 1) element1
         2) element2
      ```
      I.e. unlike BLPOP that returns a key name and one element so it uses a flat array,
      and LPOP that returns multiple elements with no key name, and again uses a flat array,
      this one has to return a nested array, and it does for for both RESP2 and RESP3 (like SCAN does)
      
      Some discuss can see: #766 #8824
      c50af0ae
  10. 14 Jun, 2021 1 commit
    • YaacovHazan's avatar
      cleanup around loadAppendOnlyFile (#9012) · 1677efb9
      YaacovHazan authored
      Today when we load the AOF on startup, the loadAppendOnlyFile checks if
      the file is openning for reading.
      This check is redundent (dead code) as we open the AOF file for writing at initServer,
      and the file will always be existing for the loadAppendOnlyFile.
      
      In this commit:
      - remove all the exit(1) from loadAppendOnlyFile, as it is the caller
        responsibility to decide what to do in case of failure.
      - move the opening of the AOF file for writing, to be after we loading it.
      - avoid return -ERR in DEBUG LOADAOF, when the AOF is existing but empty
      1677efb9
  11. 09 Jun, 2021 1 commit
    • Yossi Gottlieb's avatar
      Improve test suite to handle external servers better. (#9033) · 8a86bca5
      Yossi Gottlieb authored
      This commit revives the improves the ability to run the test suite against
      external servers, instead of launching and managing `redis-server` processes as
      part of the test fixture.
      
      This capability existed in the past, using the `--host` and `--port` options.
      However, it was quite limited and mostly useful when running a specific tests.
      Attempting to run larger chunks of the test suite experienced many issues:
      
      * Many tests depend on being able to start and control `redis-server` themselves,
      and there's no clear distinction between external server compatible and other
      tests.
      * Cluster mode is not supported (resulting with `CROSSSLOT` errors).
      
      This PR cleans up many things and makes it possible to run the entire test suite
      against an external server. It also provides more fine grained controls to
      handle cases where the external server supports a subset of the Redis commands,
      limited number of databases, cluster mode, etc.
      
      The tests directory now contains a `README.md` file that describes how this
      works.
      
      This commit also includes additional cleanups and fixes:
      
      * Tests can now be tagged.
      * Tag-based selection is now unified across `start_server`, `tags` and `test`.
      * More information is provided about skipped or ignored tests.
      * Repeated patterns in tests have been extracted to common procedures, both at a
        global level and on a per-test file basis.
      * Cleaned up some cases where test setup was based on a previous test executing
        (a major anti-pattern that repeats itself in many places).
      * Cleaned up some cases where test teardown was not part of a test (in the
        future we should have dedicated teardown code that executes even when tests
        fail).
      * Fixed some tests that were flaky running on external servers.
      8a86bca5
  12. 30 May, 2021 1 commit
    • ny0312's avatar
      Always replicate time-to-live(TTL) as absolute timestamps in milliseconds (#8474) · 53d1acd5
      ny0312 authored
      Till now, on replica full-sync we used to transfer absolute time for TTL,
      however when a command arrived (EXPIRE or EXPIREAT),
      we used to propagate it as is to replicas (possibly with relative time),
      but always translate it to EXPIREAT (absolute time) to AOF.
      
      This commit changes that and will always use absolute time for propagation.
      see discussion in #8433
      
      Furthermore, we Introduce new commands: `EXPIRETIME/PEXPIRETIME`
      that allow extracting the absolute TTL time from a key.
      53d1acd5
  13. 20 Apr, 2021 1 commit
  14. 27 Jan, 2021 1 commit
    • Raghav Muddur's avatar
      GETEX, GETDEL and SET PXAT/EXAT (#8327) · 0367a808
      Raghav Muddur authored
      This commit introduces two new command and two options for an existing command
      
      GETEX <key> [PERSIST][EX seconds][PX milliseconds] [EXAT seconds-timestamp]
      [PXAT milliseconds-timestamp]
      
      The getexCommand() function implements extended options and variants of the GET
      command. Unlike GET command this command is not read-only. Only one of the options
      can be used at a given time.
      
      1. PERSIST removes any TTL associated with the key.
      2. EX Set expiry TTL in seconds.
      3. PX Set expiry TTL in milliseconds.
      4. EXAT Same like EX instead of specifying the number of seconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      5. PXAT Same like PX instead of specifying the number of milliseconds representing the
          TTL (time to live), it takes an absolute Unix timestamp
      
      Command would return either the bulk string, error or nil.
      
      GETDEL <key>
      Would delete the key after getting.
      
      SET key value [NX] [XX] [KEEPTTL] [GET] [EX <seconds>] [PX <milliseconds>]
      [EXAT <seconds-timestamp>][PXAT <milliseconds-timestamp>]
      
      Two new options added here are EXAT and PXAT
      
      Key implementation notes
      - `SET` with `PX/EX/EXAT/PXAT` is always translated to `PXAT` in `AOF`. When relative time is
        specified (`PX/EX`), replication will always use `PX`.
      - `setexCommand` and `psetexCommand` would no longer need translation in `feedAppendOnlyFile`
        as they are modified to invoke `setGenericCommand ` with appropriate flags which will take care of
        correct AOF translation.
      - `GETEX` without any optional argument behaves like `GET`.
      - `GETEX` command is never propagated, It is either propagated as `PEXPIRE[AT], or PERSIST`.
      - `GETDEL` command is propagated as `DEL`
      - Combined the validation for `SET` and `GETEX` arguments. 
      - Test cases to validate AOF/Replication propagation
      0367a808
  15. 06 Sep, 2020 1 commit
  16. 04 May, 2020 1 commit
    • Oran Agra's avatar
      add daily github actions with libc malloc and valgrind · deee2c1e
      Oran Agra authored
      * fix memlry leaks with diskless replica short read.
      * fix a few timing issues with valgrind runs
      * fix issue with valgrind and watchdog schedule signal
      
      about the valgrind WD issue:
      the stack trace test in logging.tcl, has issues with valgrind:
      ==28808== Can't extend stack to 0x1ffeffdb38 during signal delivery for thread 1:
      ==28808==   too small or bad protection modes
      
      it seems to be some valgrind bug with SA_ONSTACK.
      SA_ONSTACK seems unneeded since WD is not recursive (SA_NODEFER was removed),
      also, not sure if it's even valid without a call to sigaltstack()
      deee2c1e
  17. 07 Oct, 2019 2 commits
  18. 22 Feb, 2017 1 commit
    • antirez's avatar
      Solaris fixes about tail usage and atomic vars. · 95883313
      antirez authored
      Testing with Solaris C compiler (SunOS 5.11 11.2 sun4v sparc sun4v)
      there were issues compiling due to atomicvar.h and running the
      tests also failed because of "tail" usage not conform with Solaris
      tail implementation. This commit fixes both the issues.
      95883313
  19. 11 Feb, 2015 1 commit
  20. 14 Dec, 2014 1 commit
    • Alon Diamant's avatar
      Added <count> parameter to SPOP: · 28802887
      Alon Diamant authored
      spopCommand() now runs spopWithCountCommand() in case the <count> param is found.
      Added intsetRandomMembers() to Intset: Copies N random members from the set into inputted 'values' array. Uses either the Knuth or Floyd sample algos depending on ratio count/size.
      Added setTypeRandomElements() to SET type: Returns a number of random elements from a non empty set. This is a version of setTypeRandomElement() that is modified in order to return multiple entries, using dictGetRandomKeys() and intsetRandomMembers().
      Added tests for SPOP with <count>: unit/type/set, unit/scripting, integration/aof
      --
      Cleaned up code a bit to match with required Redis coding style
      28802887
  21. 29 Sep, 2014 1 commit
  22. 16 Sep, 2014 1 commit
    • antirez's avatar
      Better truncated AOF loading tests. · a2c740ea
      antirez authored
      Now there are tests to write more data after loading a truncated AOF,
      testing that the loaded data is correct, appending more, and testing
      again.
      a2c740ea
  23. 08 Sep, 2014 4 commits
  24. 22 May, 2014 1 commit
  25. 19 Jun, 2013 1 commit
  26. 17 Apr, 2012 1 commit
  27. 10 Dec, 2011 1 commit
  28. 07 Dec, 2011 1 commit
  29. 11 Jul, 2011 1 commit
  30. 08 Jul, 2011 1 commit
  31. 07 Jul, 2011 1 commit
  32. 22 Apr, 2011 2 commits
  33. 01 Jul, 2010 1 commit
    • antirez's avatar
      redis.c split into many different C files. · e2641e09
      antirez authored
      networking related stuff moved into networking.c
      
      moved more code
      
      more work on layout of source code
      
      SDS instantaneuos memory saving. By Pieter and Salvatore at VMware ;)
      
      cleanly compiling again after the first split, now splitting it in more C files
      
      moving more things around... work in progress
      
      split replication code
      
      splitting more
      
      Sets split
      
      Hash split
      
      replication split
      
      even more splitting
      
      more splitting
      
      minor change
      e2641e09
  34. 02 Jun, 2010 1 commit