- 31 Mar, 2014 1 commit
-
-
antirez authored
The HyperLogLog original paper suggests using LINEARCOUNTING for cardinalities < 2.5m, however for P=14 the median / max error curves show that a value of '3' is the best pick for m = 16384.
-
- 30 Mar, 2014 2 commits
-
-
antirez authored
The more we add elements to an HyperLogLog counter, the smaller is the probability that we actually update some register. From this observation it is easy to see how it is possible to use caching of a previously computed cardinality and reuse it to serve HLLCOUNT queries as long as no register was updated in the data structure. This commit does exactly this by using just additional 8 bytes for the data structure to store a 64 bit unsigned integer value cached cardinality. When the most significant bit of the 64 bit integer is set, it means that the value computed is no longer usable since at least a single register was modified and we need to recompute it at the next call of HLLCOUNT. The value is always stored in little endian format regardless of the actual CPU endianess.
-
antirez authored
All the Redis functions that need to modify the string value of a key in a destructive way (APPEND, SETBIT, SETRANGE, ...) require to make the object unshared (if refcount > 1) and encoded in raw format (if encoding is not already REDIS_ENCODING_RAW). This was cut & pasted many times in multiple places of the code. This commit puts the small logic needed into a function called dbUnshareStringValue().
-
- 29 Mar, 2014 5 commits
-
-
antirez authored
We need to be sure that you can save a dataset in a Redis instance, reload it in a different architecture, and continue to count in the same HyperLogLog structure. So 32 and 64 bit, little or bit endian, must all guarantee to output the same hash for the same element.
-
antirez authored
The new representation is more obvious, starting from the LSB of the first byte and using bits going to MSB, and passing to next byte as needed. There was also a subtle error: first two bits were unused, everything was carried over on the right of two bits, even if it worked because of the code requirement of always having a byte more at the end. During the rewrite the code was made safer trying to avoid undefined behavior due to shifting an uint8_t for more than 8 bits.
-
antirez authored
-
antirez authored
-
antirez authored
This speedups the macros by a noticeable factor.
-
- 28 Mar, 2014 11 commits
-
-
antirez authored
-
antirez authored
There was an error in the computation of 2^register, and the sequence of zeroes computed after the hashing did not included the "1".
-
antirez authored
-
antirez authored
-
antirez authored
-
antirez authored
-
antirez authored
There was an error in the first version of the macro. Now the HLLSELFTEST test reports success.
-
antirez authored
-
antirez authored
-
antirez authored
To test the bitfield array of counters set/get macros from the Redis Tcl suite is hard, so a specialized command that is able to test the internals was developed.
-
antirez authored
-
- 27 Mar, 2014 2 commits
- 26 Mar, 2014 3 commits
-
-
antirez authored
This is not really an error but something that always happens for example when creating a new cluster, or if the sysadmin rejoins manually a node that is already known. Since useless logs don't help, moved to VERBOSE level.
-
antirez authored
New config epochs must always be obtained incrementing the currentEpoch, that is itself guaranteed to be >= the max configEpoch currently known to the node.
-
antirez authored
-
- 25 Mar, 2014 6 commits
-
-
antirez authored
Update the comment since the implementation details changed.
-
antirez authored
redis-trib used to allocate slots not considering fractions of nodes when computing the slots_per_node amount. So the fractional part was carried over till the end of the allocation, where the last node received a few more slots than any other (or a lot more if the cluster was composed of many nodes). The computation was changed to allocate slots more evenly when they are not exactly divisible for the number of masters we have.
-
antirez authored
The slave election in Redis Cluster guarantees that slaves promoted to masters always end with unique config epochs, however failures during manual reshardings, software bugs and operational errors may in theory cause two nodes to have the same configEpoch. This commit introduces a mechanism to eventually always end with different configEpochs if a collision ever happens. As a (wanted) side effect, this also ensures that after a new cluster is created, all nodes will end with a different configEpoch automatically.
-
antirez authored
-
antirez authored
Bug found by the continuous integration test running the Redis with valgrind: ==6245== Invalid read of size 8 ==6245== at 0x4C2DEEF: memcpy@GLIBC_2.2.5 (mc_replace_strmem.c:876) ==6245== by 0x41F9E6: freeMemoryIfNeeded (redis.c:3010) ==6245== by 0x41D2CC: processCommand (redis.c:2069) memmove() size argument was accounting for an extra element, going outside the bounds of the array.
-
antirez authored
In this commit: * Decrement steps are semantically differentiated from the reserved FDs. Previously both values were 32 but the meaning was different. * Make it clear that we save setrlimit errno. * Don't explicitly handle wrapping of 'f', but prevent it from happening. * Add comments to make the function flow more readable. This integrates PR #1630
-
- 24 Mar, 2014 10 commits
-
-
antirez authored
In sentinelFlushConfig() fd could be undefined when the following if statement was true: if (rewrite_status == -1) goto werr; This could cause random file descriptors to get closed.
-
Matt Stancliff authored
lookupKey() uses LRU_CLOCK(), so it seems object creation should use LRU_CLOCK() too.
-
Matt Stancliff authored
-
Matt Stancliff authored
Previously, the (!fp) would only catch lack of free space under OS X. Linux waits to discover it can't write until it actually writes contents to disk. (fwrite() returns success even if the underlying file has no free space to write into. All the errors only show up at flush/sync/close time.) Fixes antirez/redis#1604
-
Matt Stancliff authored
errno may be reset by the previous call to redisLog, so capture the original value for proper error reporting.
-
Matt Stancliff authored
Also update the original REDIS_EVENTLOOP_FDSET_INCR to include REDIS_MIN_RESERVED_FDS. REDIS_EVENTLOOP_FDSET_INCR exists to make sure more than (maxclients+RESERVED) entries are allocated, but we can only guarantee that if we include the current value of REDIS_MIN_RESERVED_FDS as a minimum for the INCR size.
-
Matt Stancliff authored
This got removed in 2e5c394f during a new feature addition. The prior commit had "break if masters.length == masters_count" but we are guaranteed to aready have that condition met since otherwise we would haven't gotten this far. Without this break statement, it's possible some masters may be forgotten and have zero replicas while other masters have more than their requested number of replicas. Thanks to carlos for pointing out this regression at: https://groups.google.com/forum/#!topic/redis-db/_WVVqDw5B7c
-
Matt Stancliff authored
This bug was introduced in 2e5c394f during a refactor. It took me a while to understand what was going on with the code, so I've refactored it further by: - Replacing boolean values with meaningful symbols - Replacing 'i' with a meaningful variable name - Adding the proper abort check - Factoring out now duplicated conditionals - Adding optional verbose logging (we're inside *four* different looping constructs, so it takes a while to figure out where all the moving parts are) - Updating comment for the section This fixes a problem when the number of master instances equaled the number of replica instances. Before, when there were equal numbers of both, nodes_count would go to zero, but the while loop would spin in i < @replicas because i would never be updated (because the nodes_list of each ip was length == 0, which triggered an endless loop of next -> i = 0 -> 0 < 1? -> true -> next -> i = 0 ...) Thanks to carlo who found this problem at: https://groups.google.com/forum/#!topic/redis-db/_WVVqDw5B7c
-
Matt Stancliff authored
Fun fact: rlim_t is an unsigned long long on all platforms. Continually subtracting from a rlim_t makes it get smaller and smaller until it wraps, then you're up to 2^64-1. This was causing an infinite loop on Redis startup if your ulimit was extremely (almost comically) low. The case of (f > oldlimit) would never be met in a case like: f = 150 while (f > 20) f -= 128 Since f is unsigned, it can't go negative and would take on values of: Iteration 1: 150 - 128 => 22 Iteration 2: 22 - 128 => 18446744073709551510 Iterations 3-∞: ... To catch the wraparound, we use the previous value of f stored in limit.rlimit_cur. If we subtract from f and get a larger number than the value it had previously, we print an error and exit since we don't have enough file descriptors to help the user at this point. Thanks to @bs3g for the inspiration to fix this problem. Patches existed from @bs3g at antirez#1227, but I needed to repair a few other parts of Redis simultaneously, so I didn't get a chance to use them.
-
Matt Stancliff authored
The log messages about open file limits have always been slightly opaque and confusing. Here's an attempt to fix their wording, detail, and meaning. Users will have a better understanding of how to fix very common problems with these reworded messages. Also, we handle a new error case when maxclients becomes less than one, essentially rendering the server unusable. We now exit on startup instead of leaving the user with a server unable to handle any connections. This fixes antirez#356 as well.
-