- 16 Jul, 2018 1 commit
-
-
Oran Agra authored
A) slave buffers didn't count internal fragmentation and sds unused space, this caused them to induce eviction although we didn't mean for it. B) slave buffers were consuming about twice the memory of what they actually needed. - this was mainly due to sdsMakeRoomFor growing to twice as much as needed each time but networking.c not storing more than 16k (partially fixed recently in 237a38737). - besides it wasn't able to store half of the new string into one buffer and the other half into the next (so the above mentioned fix helped mainly for small items). - lastly, the sds buffers had up to 30% internal fragmentation that was wasted, consumed but not used. C) inefficient performance due to starting from a small string and reallocing many times. what i changed: - creating dedicated buffers for reply list, counting their size with zmalloc_size - when creating a new reply node from, preallocate it to at least 16k. - when appending a new reply to the buffer, first fill all the unused space of the previous node before starting a new one. other changes: - expose mem_not_counted_for_evict info field for the benefit of the test suite - add a test to make sure slave buffers are counted correctly and that they don't cause eviction
-
- 03 Jul, 2018 1 commit
-
-
Jack Drogon authored
-
- 19 Jun, 2018 2 commits
-
-
antirez authored
-
antirez authored
Basically we cannot be sure that if the key is expired while writing the AOF, the main thread will surely find the key expired. There are possible race conditions like the moment at which the "now" is sampled, and the fact that time may jump backward. Think about the following: SET a 5 EXPIRE a 1 AOF rewrite starts after about 1 second. The child process finds the key expired, while in the main thread instead an INCR command is called against the key "a" immediately after a fork, and the scheduler was faster to give execution time to the main thread, so "a" is yet not expired. The main thread will generate an INCR a command to the AOF log that will be appended to the rewritten AOF file, but that INCR command will target a non existin "a" key, so a new non volatile key "a" will be created. Two observations: A) In theory by computing "now" before the fork, we should be sure that if a key is expired at that time, it will be expired later when the main thread will try to access to such key. However this does not take into account the fact that the computer time may jump backward. B) Technically we may still make the process safe by using a monotonic time source. However there were other similar related bugs, and in general the new "vision" is that Redis persistence files should represent the memory state without trying to be too smart: this makes the design more consistent, bugs less likely to arise from complex interactions, and in the end what is to fix is the Redis expire process to have less expired keys in RAM. Thanks to Oran Agra and Guy Benoish for writing me an email outlining this problem, after they conducted a Redis 5 code review.
-
- 29 May, 2018 1 commit
-
-
antirez authored
The AOF tail of a combined RDB+AOF is based on the premise of applying the AOF commands to the exact state that there was in the server while the RDB was persisted. By expiring keys while loading the RDB file, we change the state, so applying the AOF tail later may change the state. Test case: * Time1: SET a 10 * Time2: EXPIREAT a $time5 * Time3: INCR a * Time4: PERSIT A. Start bgrewiteaof with RDB preamble. The value of a is 11 without expire time. * Time5: Restart redis from the RDB+AOF: consistency violation. Thanks to @soloestoy for providing the patch. Thanks to @trevor211 for the original issue report and the initial fix. Check issue #4950 for more info.
-
- 23 Mar, 2018 1 commit
-
-
antirez authored
-
- 15 Mar, 2018 1 commit
-
-
zhaozhao.zz authored
-
- 13 Feb, 2018 2 commits
-
-
antirez authored
See #3858.
-
Guy Benoish authored
It is possible to do BGREWRITEAOF even if appendonly=no. This is by design. stopAppendonly() didn't turn off aof_rewrite_scheduled (it can be turned on again by BGREWRITEAOF even while appendonly is off anyway). After configuring `appendonly yes` it will see that the state is AOF_OFF, there's no RDB fork, so it will do rewriteAppendOnlyFileBackground() which will fail since the aof_child_pid is set (was scheduled and started by cron). Solution: stopAppendonly() will turn off the schedule flag (regardless of who asked for it). startAppendonly() will terminate any existing fork and start a new one (so it is the most recent).
-
- 15 Jan, 2018 1 commit
-
-
zhaozhao.zz authored
-
- 15 Dec, 2017 1 commit
-
-
heqin authored
-
- 14 Dec, 2017 1 commit
-
-
antirez authored
Related to #4498.
-
- 01 Dec, 2017 2 commits
- 30 Nov, 2017 2 commits
-
-
zhaozhao.zz authored
-
zhaozhao.zz authored
-
- 22 Jun, 2017 2 commits
- 16 Jun, 2017 1 commit
-
-
xuzhou authored
-
- 18 Apr, 2017 1 commit
-
-
张文康 authored
-
- 23 Feb, 2017 1 commit
-
-
oranagra authored
this commit also contains small bugfix in rdbLoadLzfStringObject a bug that currently has no implications.
-
- 20 Feb, 2017 1 commit
-
-
John.Koepi authored
-
- 09 Nov, 2016 1 commit
-
-
antirez authored
The gist of the changes is that now, partial resynchronizations between slaves and masters (without the need of a full resync with RDB transfer and so forth), work in a number of cases when it was impossible in the past. For instance: 1. When a slave is promoted to mastrer, the slaves of the old master can partially resynchronize with the new master. 2. Chained slalves (slaves of slaves) can be moved to replicate to other slaves or the master itsef, without requiring a full resync. 3. The master itself, after being turned into a slave, is able to partially resynchronize with the new master, when it joins replication again. In order to obtain this, the following main changes were operated: * Slaves also take a replication backlog, not just masters. * Same stream replication for all the slaves and sub slaves. The replication stream is identical from the top level master to its slaves and is also the same from the slaves to their sub-slaves and so forth. This means that if a slave is later promoted to master, it has the same replication backlong, and can partially resynchronize with its slaves (that were previously slaves of the old master). * A given replication history is no longer identified by the `runid` of a Redis node. There is instead a `replication ID` which changes every time the instance has a new history no longer coherent with the past one. So, for example, slaves publish the same replication history of their master, however when they are turned into masters, they publish a new replication ID, but still remember the old ID, so that they are able to partially resynchronize with slaves of the old master (up to a given offset). * The replication protocol was slightly modified so that a new extended +CONTINUE reply from the master is able to inform the slave of a replication ID change. * REPLCONF CAPA is used in order to notify masters that a slave is able to understand the new +CONTINUE reply. * The RDB file was extended with an auxiliary field that is able to select a given DB after loading in the slave, so that the slave can continue receiving the replication stream from the point it was disconnected without requiring the master to insert "SELECT" statements. This is useful in order to guarantee the "same stream" property, because the slave must be able to accumulate an identical backlog. * Slave pings to sub-slaves are now sent in a special form, when the top-level master is disconnected, in order to don't interfer with the replication stream. We just use out of band "\n" bytes as in other parts of the Redis protocol. An old design document is available here: https://gist.github.com/antirez/ae068f95c0d084891305 However the implementation is not identical to the description because during the work to implement it, different changes were needed in order to make things working well.
-
- 06 Oct, 2016 1 commit
-
-
antirez authored
It was noted by @dvirsky that it is not possible to use string functions when writing the AOF file. This sometimes is critical since the command rewriting may need to be built in the context of the AOF callback, and without access to the context, and the limited types that the AOF production functions will accept, this can be an issue. Moreover there are other needs that we can't anticipate regarding the ability to use Redis Modules APIs using the context in order to build representations to emit AOF / RDB. Because of this a new API was added that allows the user to get a temporary context from the IO context. The context is auto released if obtained when the RDB / AOF callback returns. Calling multiple time the function to get the context, always returns the same one, since it is invalid to have more than a single context.
-
- 19 Sep, 2016 2 commits
- 11 Aug, 2016 1 commit
-
-
antirez authored
-
- 09 Aug, 2016 2 commits
- 21 Jul, 2016 1 commit
-
-
antirez authored
This patch, written in collaboration with Oran Agra (@oranagra) is a companion to 780a8b1d. Together the two patches should avoid that the AOF and RDB saving processes can be spawned at the same time. Previously conditions that could lead to two saving processes at the same time were: 1. When AOF is enabled via CONFIG SET and an RDB saving process is already active. 2. When the SYNC command decides to start an RDB saving process ASAP in order to serve a new slave that cannot partially resynchronize (but only if we have a disk target for replication, for diskless replication there is not such a problem). Condition "1" is not very severe but "2" can happen often and is definitely good at degrading Redis performances in an unexpected way. The two commits have the effect of always spawning RDB savings for replication in replicationCron() instead of attempting to start an RDB save synchronously. Moreover when a BGSAVE or AOF rewrite must be performed, they are instead just postponed using flags that will try to perform such operations ASAP. Finally the BGSAVE command was modified in order to accept a SCHEDULE option so that if an AOF rewrite is in progress, when this option is given, the command no longer returns an error, but instead schedules an RDB rewrite operation for when it will be possible to start it.
-
- 03 Jun, 2016 1 commit
-
-
antirez authored
-
- 25 Apr, 2016 1 commit
-
-
Oran Agra authored
-
- 15 Feb, 2016 1 commit
-
-
antirez authored
Close #3086.
-
- 17 Nov, 2015 1 commit
-
-
antirez authored
-
- 13 Nov, 2015 1 commit
-
-
antirez authored
-
- 01 Oct, 2015 2 commits
- 27 Jul, 2015 2 commits
- 26 Jul, 2015 1 commit
-
-
antirez authored
-