- 31 May, 2022 1 commit
-
-
DarrenJiang13 authored
The amount of `server.stat_net_output_bytes/server.stat_net_input_bytes` is actually the sum of replication flow and users' data flow. It may cause confusions like this: "Why does my server get such a large output_bytes while I am doing nothing? ". After discussions and revisions, now here is the change about what this PR brings (final version before merge): - 2 server variables to count the network bytes during replication, including fullsync and propagate bytes. - `server.stat_net_repl_output_bytes`/`server.stat_net_repl_input_bytes` - 3 info fields to print the input and output of repl bytes and instantaneous value of total repl bytes. - `total_net_repl_input_bytes` / `total_net_repl_output_bytes` - `instantaneous_repl_total_kbps` - 1 new API `rioCheckType()` to check the type of rio. So we can use this to distinguish between diskless and diskbased replication - 2 new counting items to keep network statistics consistent between master and slave - rdb portion during diskless replica. in `rdbLoadProgressCallback()` - first line of the full sync payload. in `readSyncBulkPayload()` Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 31 Dec, 2021 1 commit
-
-
sundb authored
Now if redis is still loading when we receive sigterm, we will wait for the loading to reach the event loop (once in 2mb) before actually shutting down. See #10003. This change caused valgrind CI to fail. See https://github.com/redis/redis/runs/4662901673?check_suite_focus=true This pr is mainly to solve the problem that redis process cannot be exited normally. When the master is disconnected, if repl is processing diskless loading and using `connRead` to read data from master, it may enter an infinite retry state, which does not handle `connRead` returning 0(master connection disconnected).
-
- 04 Nov, 2021 1 commit
-
-
menwen authored
When repl-diskless-load is enabled, the connection is set to the blocking state. The connection may be interrupted by a signal during a system call. This would have resulted in a disconnection and possibly a reconnection loop. Co-authored-by:
Oran Agra <oran@redislabs.com>
-
- 30 Aug, 2021 1 commit
-
-
Wang Yuan authored
We implement incremental data sync in rio.c by call fsync, on slow disk, that may cost a lot of time, sync_file_range could provide async fsync, so we could serialize key/value and sync file data at the same time. > one tip for sync_file_range usage: http://lkml.iu.edu/hypermail/linux/kernel/1005.2/01845.html Additionally, this change avoids a single large write to be used, which can result in a mass of dirty pages in the kernel (increasing the risk of someone else's write to block). On HDD, current solution could reduce approximate half of dumping RDB time, this PR costs 50s for dump 7.7G rdb but unstable branch costs 93s. On NVME SSD, this PR can't reduce much time, this PR costs 40s, unstable branch costs 48s. Moreover, I find calling data sync every 4MB is better than 32MB.
-
- 30 Jul, 2021 1 commit
-
-
Ewg-c authored
minor refactoring for rioConnRead and adding errno
-
- 10 Jun, 2021 1 commit
-
-
Binbin authored
This PR adds a spell checker CI action that will fail future PRs if they introduce typos and spelling mistakes. This spell checker is based on blacklist of common spelling mistakes, so it will not catch everything, but at least it is also unlikely to cause false positives. Besides that, the PR also fixes many spelling mistakes and types, not all are a result of the spell checker we use. Here's a summary of other changes: 1. Scanned the entire source code and fixes all sorts of typos and spelling mistakes (including missing or extra spaces). 2. Outdated function / variable / argument names in comments 3. Fix outdated keyspace masks error log when we check `config.notify-keyspace-events` in loadServerConfigFromString. 4. Trim the white space at the end of line in `module.c`. Check: https://github.com/redis/redis/pull/7751 5. Some outdated https link URLs. 6. Fix some outdated comment. Such as: - In README: about the rdb, we used to said create a `thread`, change to `process` - dbRandomKey function coment (about the dictGetRandomKey, change to dictGetFairRandomKey) - notifyKeyspaceEvent fucntion comment (add type arg) - Some others minor fix in comment (Most of them are incorrectly quoted by variable names) 7. Modified the error log so that users can easily distinguish between TCP and TLS in `changeBindAddr`
-
- 01 Apr, 2021 1 commit
-
-
Wang Yuan authored
In `aof.c`, we call fsync when stop aof, and now print a log to let user know that if fail. In `cluster.c`, we now return error, the calling function already handles these write errors. In `redis-cli.c`, users hope to save rdb, we now print a message if fsync failed. In `rio.c`, we now treat fsync errors like we do for write errors. In `server.c`, we try to fsync aof file when shutdown redis, we only can print one log if fail. In `bio.c`, if failing to fsync aof file, we will set `aof_bio_fsync_status` to error , and reject writing just like last writing aof error, moreover also set INFO command field `aof_last_write_status` to error.
-
- 14 Mar, 2021 1 commit
-
-
Huang Zhw authored
-
- 24 Jul, 2020 1 commit
-
-
zhaozhao.zz authored
-
- 23 Jul, 2020 1 commit
-
-
Oran Agra authored
this code is in use only if the master is disk-based, and the replica is diskless. In this case we use a buffered reader, but we must avoid reading past the rdb file, into the command stream. which Luckly rdb.c doesn't really attempt to do (it knows how much it should read). When rioConnRead detects that the extra buffering attempt reaches beyond the read limit it should read less, but if the caller actually requested more, then it should return with an error rather than a short read. the bug would have resulted in short read. in order to fix it, the code must consider the real requested size, and not the extra buffering size.
-
- 07 Oct, 2019 2 commits
-
-
Oran Agra authored
misc: - handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents - fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed) - add key-load-delay config for testing - trim connShutdown which is no longer needed - rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs - don't detect rdb child exited (don't call wait3) until we detect the pipe is closed - Cleanup bad optimization from rio.c, add another one
-
Yossi Gottlieb authored
* Introduce a connection abstraction layer for all socket operations and integrate it across the code base. * Provide an optional TLS connections implementation based on OpenSSL. * Pull a newer version of hiredis with TLS support. * Tests, redis-cli updates for TLS support.
-
- 17 Jul, 2019 1 commit
-
-
antirez authored
-
- 10 Jul, 2019 1 commit
-
-
antirez authored
-
- 08 Jul, 2019 3 commits
-
-
antirez authored
-
antirez authored
-
Oran Agra authored
The implementation of the diskless replication was currently diskless only on the master side. The slave side was still storing the received rdb file to the disk before loading it back in and parsing it. This commit adds two modes to load rdb directly from socket: 1) when-empty 2) using "swapdb" the third mode of using diskless slave by flushdb is risky and currently not included. other changes: -------------- distinguish between aof configuration and state so that we can re-enable aof only when sync eventually succeeds (and not when exiting from readSyncBulkPayload after a failed attempt) also a CONFIG GET and INFO during rdb loading would have lied When loading rdb from the network, don't kill the server on short read (that can be a network error) Fix rdb check when performed on preamble AOF tests: run replication tests for diskless slave too make replication test a bit more aggressive Add test for diskless load swapdb
-
- 15 Mar, 2018 1 commit
-
-
zhaozhao.zz authored
-
- 29 Dec, 2017 1 commit
-
-
Oran Agra authored
- protocol parsing (processMultibulkBuffer) was limitted to 32big positions in the buffer readQueryFromClient potential overflow - rioWriteBulkCount used int, although rioWriteBulkString gave it size_t - several places in sds.c that used int for string length or index. - bugfix in RM_SaveAuxField (return was 1 or -1 and not length) - RM_SaveStringBuffer was limitted to 32bit length
-
- 25 Apr, 2016 1 commit
-
-
Oran Agra authored
-
- 27 Jul, 2015 1 commit
-
-
antirez authored
-
- 26 Jul, 2015 2 commits
- 22 Oct, 2014 1 commit
-
-
antirez authored
EWOULDBLOCK with the fdset rio target is returned when we try to write but the send timeout socket option triggered an error. Better to translate the error in something the user can actually recognize as a timeout.
-
- 17 Oct, 2014 3 commits
-
-
antirez authored
-
antirez authored
While the socket is set in blocking mode, we still can get short writes writing to a socket.
-
antirez authored
To perform a socket write() for each RDB rio API write call was extremely unefficient, so now rio has minimal buffering capabilities. Writes are accumulated into a buffer and only when a given limit is reacehd are actually wrote to the N slaves FDs. Trivia: rio lacked support for buffering since our targets were: 1) Memory buffers. 2) C standard I/O. Both were buffered already.
-
- 14 Oct, 2014 1 commit
-
-
antirez authored
Fdset target is used when we want to write an RDB file directly to slave's sockets. In this setup as long as there is a single slave that is still receiving our payload, we want to continue sennding instead of aborting. However rio calls should abort of no FD is ok. Also we want the errors reported so that we can signal the parent who is ok and who is broken, so there is a new set integers with the state of each fd. Zero is ok, non-zero is the errno of the failure, if avaialble, or a generic EIO.
-
- 10 Oct, 2014 2 commits
- 22 Jan, 2014 1 commit
-
-
antirez authored
Incremental flushing in rio.c is only used to avoid huge kernel buffers synched to slow disks creating big latency spikes, so this fix has no durability implications, however it is certainly more correct to make sure that the FILE buffers are flushed to the kernel before calling fsync on the file descriptor. Thanks to Li Shao Kai for reporting this issue in the Redis mailing list.
-
- 16 Jul, 2013 1 commit
-
-
yoav authored
-
- 24 Apr, 2013 1 commit
-
-
antirez authored
-
- 29 Nov, 2012 1 commit
-
-
antirez authored
The idea is to be able to identify a build in a unique way, so for instance after a bug report we can recognize that the build is the one of a popular Linux distribution and perform the debugging in the same environment.
-
- 08 Nov, 2012 1 commit
-
-
antirez authored
-
- 09 Apr, 2012 3 commits
- 23 Oct, 2011 1 commit
-
-
antirez authored
-
- 22 Sep, 2011 1 commit
-
-
antirez authored
rioInitWithFile nad rioInitWithBuffer functions now take a rio structure pointer to avoid copying a structure to return value to the caller.
-