Unverified Commit 87789fae authored by chenyang8094's avatar chenyang8094 Committed by GitHub
Browse files

Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788)



Implement Multi-Part AOF mechanism to avoid overheads during AOFRW.
Introducing a folder with multiple AOF files tracked by a manifest file.

The main issues with the the original AOFRW mechanism are:
* buffering of commands that are processed during rewrite (consuming a lot of RAM)
* freezes of the main process when the AOFRW completes to drain the remaining part of the buffer and fsync it.
* double disk IO for the data that arrives during AOFRW (had to be written to both the old and new AOF files)

The main modifications of this PR:
1. Remove the AOF rewrite buffer and related code.
2. Divide the AOF into multiple files, they are classified as two types, one is the the `BASE` type,
  it represents the full amount of data (Maybe AOF or RDB format) after each AOFRW, there is only
  one `BASE` file at most. The second is `INCR` type, may have more than one. They represent the
  incremental commands since the last AOFRW.
3. Use a AOF manifest file to record and manage these AOF files mentioned above.
4. The original configuration of `appendfilename` will be the base part of the new file name, for example:
  `appendonly.aof.1.base.rdb` and `appendonly.aof.2.incr.aof`
5. Add manifest-related TCL tests, and modified some existing tests that depend on the `appendfilename`
6. Remove the `aof_rewrite_buffer_length` field in info.
7. Add `aof-disable-auto-gc` configuration. By default we're automatically deleting HISTORY type AOFs.
  It also gives users the opportunity to preserve the history AOFs. just for testing use now.
8. Add AOFRW limiting measure. When the AOFRW failures reaches the threshold (3 times now),
  we will delay the execution of the next AOFRW by 1 minute. If the next AOFRW also fails, it will be
  delayed by 2 minutes. The next is 4, 8, 16, the maximum delay is 60 minutes (1 hour). During the limit
  period, we can still use the 'bgrewriteaof' command to execute AOFRW immediately.
9. Support upgrade (load) data from old version redis.
10. Add `appenddirname` configuration, as the directory name of the append only files. All AOF files and
  manifest file will be placed in this directory.
11. Only the last AOF file (BASE or INCR) can be truncated. Otherwise redis will exit even if
  `aof-load-truncated` is enabled.
Co-authored-by: default avatarOran Agra <oran@redislabs.com>
parent 78a62c01
...@@ -30,10 +30,16 @@ proc clean_persistence config { ...@@ -30,10 +30,16 @@ proc clean_persistence config {
# we may wanna keep the logs for later, but let's clean the persistence # we may wanna keep the logs for later, but let's clean the persistence
# files right away, since they can accumulate and take up a lot of space # files right away, since they can accumulate and take up a lot of space
set config [dict get $config "config"] set config [dict get $config "config"]
set rdb [format "%s/%s" [dict get $config "dir"] "dump.rdb"] set dir [dict get $config "dir"]
set aof [format "%s/%s" [dict get $config "dir"] "appendonly.aof"] set rdb [format "%s/%s" $dir "dump.rdb"]
if {[dict exists $config "appenddirname"]} {
set aofdir [dict get $config "appenddirname"]
} else {
set aofdir "appendonlydir"
}
set aof_dirpath [format "%s/%s" $dir $aofdir]
clean_aof_persistence $aof_dirpath
catch {exec rm -rf $rdb} catch {exec rm -rf $rdb}
catch {exec rm -rf $aof}
} }
proc kill_server config { proc kill_server config {
......
...@@ -84,7 +84,7 @@ proc status {r property} { ...@@ -84,7 +84,7 @@ proc status {r property} {
proc waitForBgsave r { proc waitForBgsave r {
while 1 { while 1 {
if {[status r rdb_bgsave_in_progress] eq 1} { if {[status $r rdb_bgsave_in_progress] eq 1} {
if {$::verbose} { if {$::verbose} {
puts -nonewline "\nWaiting for background save to finish... " puts -nonewline "\nWaiting for background save to finish... "
flush stdout flush stdout
...@@ -98,7 +98,7 @@ proc waitForBgsave r { ...@@ -98,7 +98,7 @@ proc waitForBgsave r {
proc waitForBgrewriteaof r { proc waitForBgrewriteaof r {
while 1 { while 1 {
if {[status r aof_rewrite_in_progress] eq 1} { if {[status $r aof_rewrite_in_progress] eq 1} {
if {$::verbose} { if {$::verbose} {
puts -nonewline "\nWaiting for background AOF rewrite to finish... " puts -nonewline "\nWaiting for background AOF rewrite to finish... "
flush stdout flush stdout
......
...@@ -6,6 +6,7 @@ package require Tcl 8.5 ...@@ -6,6 +6,7 @@ package require Tcl 8.5
set tcl_precision 17 set tcl_precision 17
source tests/support/redis.tcl source tests/support/redis.tcl
source tests/support/aofmanifest.tcl
source tests/support/server.tcl source tests/support/server.tcl
source tests/support/tmpfile.tcl source tests/support/tmpfile.tcl
source tests/support/test.tcl source tests/support/test.tcl
...@@ -46,6 +47,7 @@ set ::all_tests { ...@@ -46,6 +47,7 @@ set ::all_tests {
integration/replication-buffer integration/replication-buffer
integration/shutdown integration/shutdown
integration/aof integration/aof
integration/aof-multi-part
integration/rdb integration/rdb
integration/corrupt-dump integration/corrupt-dump
integration/corrupt-dump-fuzzer integration/corrupt-dump-fuzzer
......
...@@ -308,14 +308,14 @@ start_server {tags {"expire"}} { ...@@ -308,14 +308,14 @@ start_server {tags {"expire"}} {
} {-2} } {-2}
# Start a new server with empty data and AOF file. # Start a new server with empty data and AOF file.
start_server {overrides {appendonly {yes} appendfilename {appendonly.aof} appendfsync always} tags {external:skip}} { start_server {overrides {appendonly {yes} appendfsync always} tags {external:skip}} {
test {All time-to-live(TTL) in commands are propagated as absolute timestamp in milliseconds in AOF} { test {All time-to-live(TTL) in commands are propagated as absolute timestamp in milliseconds in AOF} {
# This test makes sure that expire times are propagated as absolute # This test makes sure that expire times are propagated as absolute
# times to the AOF file and not as relative time, so that when the AOF # times to the AOF file and not as relative time, so that when the AOF
# is reloaded the TTLs are not being shifted forward to the future. # is reloaded the TTLs are not being shifted forward to the future.
# We want the time to logically pass when the server is restarted! # We want the time to logically pass when the server is restarted!
set aof [file join [lindex [r config get dir] 1] [lindex [r config get appendfilename] 1]] set aof [get_last_incr_aof_path r]
# Apply each TTL-related command to a unique key # Apply each TTL-related command to a unique key
# SET commands # SET commands
......
...@@ -183,6 +183,7 @@ start_server {tags {"introspection"}} { ...@@ -183,6 +183,7 @@ start_server {tags {"introspection"}} {
pidfile pidfile
syslog-ident syslog-ident
appendfilename appendfilename
appenddirname
supervised supervised
syslog-facility syslog-facility
databases databases
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment