- 08 Jan, 2015 2 commits
-
-
antirez authored
-
antirez authored
This commit introduces a new RDB data type called 'aux'. It is used in order to insert inside an RDB file key-value pairs that may serve different needs, without breaking backward compatibility when new informations are embedded inside an RDB file. The contract between Redis versions is to ignore unknown aux fields when encountered. Aux fields can be used in order to: 1. Augment the RDB file with info like version of Redis that created the RDB file, creation time, used memory while the RDB was created, and so forth. 2. Add state about Redis inside the RDB file that we need to reload later: replication offset, previos master run ID, in order to improve failovers safety and allow partial resynchronization after a slave restart. 3. Anything that we may want to add to RDB files without breaking the ability of past versions of Redis to load the file.
-
- 07 Jan, 2015 4 commits
-
-
antirez authored
-
antirez authored
The new opcode is an hint about the size of the dataset (keys and number of expires) we are going to load for a given Redis database inside the RDB file. Since hash tables are resized accordingly ASAP, useless rehashing is avoided, speeding up load times significantly, in the order of ~ 20% or more for larger data sets. Related issue: #1719
-
antirez authored
-
antirez authored
Before we needed to create a string object with an embedded SDS, adn basically duplicate the SDS part into a plain zmalloc() allocation.
-
- 24 Dec, 2014 1 commit
-
-
antirez authored
-
- 23 Dec, 2014 33 commits
-
-
Matt Stancliff authored
This is lzf_c and lzf_d from http://dist.schmorp.de/liblzf/liblzf-3.6.tar.gz
-
Matt Stancliff authored
This also defines REDIS_STATIC='' for building everything inside src/ and everything inside deps/lua/.
-
Matt Stancliff authored
Adds: ql_compressed (boolean, 1 if compression enabled for list, 0 otherwise) Adds: ql_uncompressed_size (actual uncompressed size of all quicklistNodes) Adds: ql_ziplist_max (quicklist max ziplist fill factor) Compression ratio of the list is then ql_uncompressed_size / serializedlength We report ql_uncompressed_size for all quicklists because serializedlength is a _compressed_ representation anyway. Sample output from a large list: 127.0.0.1:6379> llen abc (integer) 38370061 127.0.0.1:6379> debug object abc Value at:0x7ff97b51d140 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718164 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:0 ql_uncompressed_size:1643187761 (1.36s) The 1.36s result time is because rdbSavedObjectLen() is serializing the object, not because of any new stats reporting. If we run DEBUG OBJECT on a compressed list, DEBUG OBJECT takes almost *zero* time because rdbSavedObjectLen() reuses already-compressed ziplists: 127.0.0.1:6379> debug object abc Value at:0x7fe5c5800040 refcount:1 encoding:quicklist serializedlength:19878335 lru:9718109 lru_seconds_idle:5 ql_nodes:21945 ql_avg_node:1748.46 ql_ziplist_max:-2 ql_compressed:1 ql_uncompressed_size:1643187761
-
Matt Stancliff authored
This removes: - list-max-ziplist-entries - list-max-ziplist-value This adds: - list-max-ziplist-size - list-compress-depth Also updates config file with new sections and updates tests to use quicklist settings instead of old list settings.
-
Matt Stancliff authored
Actually makes a noticeable difference. Branch hints were selected based on profiler hotspots.
-
Matt Stancliff authored
Let user set how many nodes to *not* compress. We can specify a compression "depth" of how many nodes to leave uncompressed on each end of the quicklist. Depth 0 = disable compression. Depth 1 = only leave head/tail uncompressed. - (read as: "skip 1 node on each end of the list before compressing") Depth 2 = leave head, head->next, tail->prev, tail uncompressed. - ("skip 2 nodes on each end of the list before compressing") Depth 3 = Depth 2 + head->next->next + tail->prev->prev - ("skip 3 nodes...") etc. This also: - updates RDB storage to use native quicklist compression (if node is already compressed) instead of uncompressing, generating the RDB string, then re-compressing the quicklist node. - internalizes the "fill" parameter for the quicklist so we don't need to pass it to _every_ function. Now it's just a property of the list. - allows a runtime-configurable compression option, so we can expose a compresion parameter in the configuration file if people want to trade slight request-per-second performance for up to 90%+ memory savings in some situations. - updates the quicklist tests to do multiple passes: 200k+ tests now.
-
Matt Stancliff authored
Added field 'ql_nodes' and 'ql_avg_per_node'. ql_nodes is the number of quicklist nodes in the quicklist. ql_avg_node is the average fill level in each quicklist node. (LLEN / QL_NODES) Sample output: 127.0.0.1:6379> DEBUG object b Value at:0x7fa42bf2fed0 refcount:1 encoding:quicklist serializedlength:18489 lru:8983768 lru_seconds_idle:3 ql_nodes:430 ql_avg_per_node:511.73 127.0.0.1:6379> llen b (integer) 220044
-
Matt Stancliff authored
We trust zmalloc to kill the whole process on memory failure
-
Matt Stancliff authored
Previously, the old test ran 5,000 loops and used about 500k. With quicklist, storing those same 5,000 loops takes up 24k, so the "large value check" failed! This increases the test to 20,000 loops which makes the object dump 96k.
-
Matt Stancliff authored
Turns out it's a huge improvement during save/reload/migrate/restore because, with compression enabled, we're compressing 4k or 8k chunks of data consisting of multiple elements in one ziplist instead of compressing series of smaller individual elements.
-
Matt Stancliff authored
This saves us an unnecessary zmalloc, memcpy, and two frees.
-
Matt Stancliff authored
Use the existing memory space for an SDS to convert it to a regular character buffer so we don't need to allocate duplicate space just to extract a usable buffer for native operations.
-
Matt Stancliff authored
Fill factor now has two options: - negative (1-5) for size-based ziplist filling - positive for length-based ziplist filling with implicit size cap. Negative offsets define ziplist size limits of: -1: 4k -2: 8k -3: 16k -4: 32k -5: 64k Positive offsets now automatically limit their max size to 8k. Any elements larger than 8k will be in individual nodes. Positive ziplist fill factors will keep adding elements to a ziplist until one of: - ziplist has FILL number of elements - or - - ziplist grows above our ziplist max size (currently 8k) When using positive fill factors, if you insert a large element (over 8k), that element will automatically allocate an individual quicklist node with one element and no other elements will be in the same ziplist inside that quicklist node. When using negative fill factors, elements up to the size limit can be added to one quicklist node. If an element is added larger than the max ziplist size, that element will be allocated an individual ziplist in a new quicklist node. Tests also updated to start testing at fill factor -5.
-
Matt Stancliff authored
-
Matt Stancliff authored
Freeing our test lists helps keep valgrind output clean
-
Matt Stancliff authored
This started out as #2158 by sunheehnus, but I kept rewriting it until I could understand things more easily and get a few more correctness guarantees out of the readability flow. The original commit created and returned a new ziplist with the contents of both input ziplists, but I prefer to grow one of the input ziplists and destroy the other one. So, instead of malloc+copy as in #2158, the merge now reallocs one of the existing ziplists and copies the other ziplist into the new space. Also added merge test cases to ziplistTest()
-
Matt Stancliff authored
This replaces individual ziplist vs. linkedlist representations for Redis list operations. Big thanks for all the reviews and feedback from everybody in https://github.com/antirez/redis/pull/2143
-
Matt Stancliff authored
-
Matt Stancliff authored
Valgrind can't detect 'memset' initializes things, so let's statically initialize them to remove some unnecessary warnings.
-
Matt Stancliff authored
The previous test wasn't returning the new ziplist, so the test was invalid. Now the test works properly. These problems were simultaenously discovered in #2154 and that PR also had an additional fix we included here.
-
Matt Stancliff authored
It's valid to delete from negative offsets, so we *don't* want unsigned arguments here.
-
Matt Stancliff authored
zipEntry was returning a struct, but that caused some problems with tests under 32 bit builds. The tests run better if we operate on structs allocated in the caller without worrying about copying on return.
-
Matt Stancliff authored
-
Matt Stancliff authored
-
Matt Stancliff authored
Previously, many files had individual main() functions for testing, but each required being compiled with their own testing flags. That gets difficult when you have 8 different flags you need to set just to run all tests (plus, some test files required other files to be compiled aaginst them, and it seems some didn't build at all without including the rest of Redis). Now all individual test main() funcions are renamed to a test function for the file itself and one global REDIS_TEST define enables testing across the entire codebase. Tests can now be run with: - `./redis-server test <test>` e.g. ./redis-server test ziplist If REDIS_TEST is not defined, then no tests get included and no tests are included in the final redis-server binary.
-
Matt Stancliff authored
Only happen when compiled with the test define.
-
Matt Stancliff authored
Uses jemalloc function malloc_stats_print() to return stats about what jemalloc has allocated internally.
-
Matt Stancliff authored
Refactor a common pattern into one function so we don't end up with copy/paste programming.
-
antirez authored
-
antirez authored
1. Server unxtime may remain not updated while loading AOF, so ETA is not updated correctly. 2. Number of processed byte was not initialized. 3. Possible division by zero condition (likely cause of issue #1932).
-
Salvatore Sanfilippo authored
Improve redis-trib replica assignment
-
Salvatore Sanfilippo authored
Add 'age' value to SENTINEL INFO-CACHE
-
Matt Stancliff authored
-