1. 11 Nov, 2021 1 commit
    • Ozan Tezcan's avatar
      Add sanitizer support and clean up sanitizer findings (#9601) · b91d8b28
      Ozan Tezcan authored
      - Added sanitizer support. `address`, `undefined` and `thread` sanitizers are available.  
      - To build Redis with desired sanitizer : `make SANITIZER=undefined`
      - There were some sanitizer findings, cleaned up codebase
      - Added tests with address and undefined behavior sanitizers to daily CI.
      - Added tests with address sanitizer to the per-PR CI (smoke out mem leaks sooner).
      
      Basically, there are three types of issues : 
      
      **1- Unaligned load/store** : Most probably, this issue may cause a crash on a platform that
      does not support unaligned access. Redis does unaligned access only on supported platforms.
      
      **2- Signed integer overflow.** Although, signed overflow issue can be problematic time to time
      and change how compiler generates code, current findings mostly about signed shift or simple
      addition overflow. For most platforms Redis can be compiled for, this wouldn't cause any issue
      as far as I can tell (checked generated code on godbolt.org).
      
       **3 -Minor leak** (redis-cli), **use-after-free**(just before calling exit());
      
      UB means nothing guaranteed and risky to reason about program behavior but I don't think any
      of the fixes here worth backporting. As sanitizers are now part of the CI, preventing new issues
      will be the real benefit. 
      b91d8b28
  2. 31 Aug, 2021 1 commit
    • Viktor Söderqvist's avatar
      Slot-to-keys using dict entry metadata (#9356) · f24c63a2
      Viktor Söderqvist authored
      
      
      * Enhance dict to support arbitrary metadata carried in dictEntry
      Co-authored-by: default avatarViktor Söderqvist <viktor.soderqvist@est.tech>
      
      * Rewrite slot-to-keys mapping to linked lists using dict entry metadata
      
      This is a memory enhancement for Redis Cluster.
      
      The radix tree slots_to_keys (which duplicates all key names prefixed with their
      slot number) is replaced with a linked list for each slot. The dict entries of
      the same cluster slot form a linked list and the pointers are stored as metadata
      in each dict entry of the main DB dict.
      
      This commit also moves the slot-to-key API from db.c to cluster.c.
      Co-authored-by: default avatarJim Brunner <brunnerj@amazon.com>
      f24c63a2
  3. 15 Aug, 2021 1 commit
  4. 05 Aug, 2021 1 commit
    • yoav-steinberg's avatar
      dict struct memory optimizations (#9228) · 5e908a29
      yoav-steinberg authored
      Reduce dict struct memory overhead
      on 64bit dict size goes down from jemalloc's 96 byte bin to its 56 byte bin.
      
      summary of changes:
      - Remove `privdata` from callbacks and dict creation. (this affects many files, see "Interface change" below).
      - Meld `dictht` struct into the `dict` struct to eliminate struct padding. (this affects just dict.c and defrag.c)
      - Eliminate the `sizemask` field, can be calculated from size when needed.
      - Convert the `size` field into `size_exp` (exponent), utilizes one byte instead of 8.
      
      Interface change: pass dict pointer to dict type call back functions.
      This is instead of passing the removed privdata field. In the future if
      we'd like to have private data in the callbacks we can extract it from
      the dict type. We can extend dictType to include a custom dict struct
      allocator and use it to allocate more data at the end of the dict
      struct. This data can then be used to store private data later acccessed
      by the callbacks.
      5e908a29
  5. 09 May, 2021 1 commit
  6. 10 Mar, 2021 1 commit
    • sundb's avatar
      Add run all test support with define REDIS_TEST (#8570) · 95d6297d
      sundb authored
      1. Add `redis-server test all` support to run all tests.
      2. Add redis test to daily ci.
      3. Add `--accurate` option to run slow tests for more iterations (so that
         by default we run less cycles (shorter time, and less prints).
      4. Move dict benchmark to REDIS_TEST.
      5. fix some leaks in tests
      6. make quicklist tests run on a specific fill set of options rather than huge ranges
      7. move some prints in quicklist test outside their loops to reduce prints
      8. removing sds.h from dict.c since it is now used in both redis-server and
         redis-cli (uses hiredis sds)
      95d6297d
  7. 20 Feb, 2021 1 commit
    • Jim Brunner's avatar
      dict: pause rehash, minor readability refactor (#8515) · 06966d2a
      Jim Brunner authored
      The `dict` field `iterators` is misleading and incorrect. 
      This variable is used for 1 purpose - to pause rehashing.
      
      The current `iterators` field doesn't actually count "iterators".
      It counts "safe iterators".  But - it doesn't actually count safe iterators
      either.  For one, it's only incremented once the safe iterator begins to
      iterate, not when it's created.  It's also incremented in `dictScan` to
      prevent rehashing (and commented to make it clear why `iterators` is
      being incremented during a scan).
      
      This update renames the field as `pauserehash` and creates 2 helper
      macros `dictPauseRehashing(d)` and `dictResumeRehashing(d)`
      06966d2a
  8. 23 Dec, 2020 1 commit
    • Greg Femec's avatar
      Fix random element selection for large hash tables. (#8133) · 266949c7
      Greg Femec authored
      
      
      When a database on a 64 bit build grows past 2^31 keys, the underlying hash table expands to 2^32 buckets. After this point, the algorithms for selecting random elements only return elements from half of the available buckets because they use random() which has a range of 0 to 2^31 - 1. This causes problems for eviction policies which use dictGetSomeKeys or dictGetRandomKey. Over time they cause the hash table to become unbalanced because, while new keys are spread out evenly across all buckets, evictions come from only half of the available buckets. Eventually this half of the table starts to run out of keys and it takes longer and longer to find candidates for eviction. This continues until no more evictions can happen.
      
      This solution addresses this by using a 64 bit PRNG instead of libc random().
      Co-authored-by: default avatarGreg Femec <gfemec@google.com>
      266949c7
  9. 06 Dec, 2020 2 commits
    • Oran Agra's avatar
      Sanitize dump payload: fail RESTORE if memory allocation fails · 7ca00d69
      Oran Agra authored
      When RDB input attempts to make a huge memory allocation that fails,
      RESTORE should fail gracefully rather than die with panic
      7ca00d69
    • Wang Yuan's avatar
      Limit the main db and expires dictionaries to expand (#7954) · 75f9dec6
      Wang Yuan authored
      As we know, redis may reject user's requests or evict some keys if
      used memory is over maxmemory. Dictionaries expanding may make
      things worse, some big dictionaries, such as main db and expires dict,
      may eat huge memory at once for allocating a new big hash table and be
      far more than maxmemory after expanding.
      There are related issues: #4213 #4583
      
      More details, when expand dict in redis, we will allocate a new big
      ht[1] that generally is double of ht[0], The size of ht[1] will be
      very big if ht[0] already is big. For db dict, if we have more than
      64 million keys, we need to cost 1GB for ht[1] when dict expands.
      
      If the sum of used memory and new hash table of dict needed exceeds
      maxmemory, we shouldn't allow the dict to expand. Because, if we
      enable keys eviction, we still couldn't add much more keys after
      eviction and rehashing, what's worse, redis will keep less keys when
      redis only remains a little memory for storing new hash table instead
      of users' data. Moreover users can't write data in redis if disable
      keys eviction.
      
      What this commit changed ?
      
      Add a new member function expandAllowed for dict type, it provide a way
      for caller to allow expand or not. We expose two parameters for this
      function: more memory needed for expanding and dict current load factor,
      users can implement a function to make a decision by them.
      For main db dict and expires dict type, these dictionaries may be very
      big and cost huge memory for expanding, so we implement a judgement
      function: we can stop dict to expand provisionally if used memory will
      be over maxmemory after dict expands, but to guarantee the performance
      of redis, we still allow dict to expand if dict load factor exceeds the
      safe load factor.
      Add test cases to verify we don't allow main db to expand when left
      memory is not enough, so that avoid keys eviction.
      
      Other changes:
      
      For new hash table size when expand. Before this commit, the size is
      that double used of dict and later _dictNextPower. Actually we aim to
      control a dict load factor between 0.5 and 1.0. Now we replace *2 with
      +1, since the first check is that used >= size, the outcome of before
      will usually be the same as _dictNextPower(used+1). The only case where
      it'll differ is when dict_can_resize is false during fork, so that later
      the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e.
      _dictNextPower(1025*2) will return 4096).
      Fix rehash test cases due to changing algorithm of new hash table size
      when expand.
      75f9dec6
  10. 18 Feb, 2019 1 commit
  11. 05 Dec, 2017 1 commit
  12. 20 Feb, 2017 1 commit
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · adeed29a
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      adeed29a
  13. 02 Jan, 2017 1 commit
  14. 30 Dec, 2016 1 commit
  15. 14 Sep, 2016 2 commits
    • antirez's avatar
      dict.c: dictReplaceRaw() -> dictAddOrFind(). · 09a50d34
      antirez authored
      What they say about "naming things" in programming?
      09a50d34
    • oranagra's avatar
      dict.c: introduce dictUnlink(). · afcbcc0e
      oranagra authored
      Notes by @antirez:
      
      This patch was picked from a larger commit by Oran and adapted to change
      the API a bit. The basic idea is to avoid double lookups when there is
      to use the value of the deleted entry.
      
      BEFORE:
      
          entry = dictFind( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictDelete(...);         /* 2nd lookup. */
      
      AFTER:
      
          entry = dictUnlink( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictFreeUnlinkedEntry(entry); /* No lookups!. */
      afcbcc0e
  16. 12 Sep, 2016 1 commit
  17. 01 Oct, 2015 1 commit
  18. 14 Jul, 2015 1 commit
    • antirez's avatar
      DEBUG HTSTATS <dbid> added. · 0f64080d
      antirez authored
      The command reports information about the hash table internal state
      representing the specified database ID.
      
      This can be used in order to investigate rehashings, memory usage issues
      and for other debugging purposes.
      0f64080d
  19. 11 Feb, 2015 2 commits
  20. 26 Aug, 2014 1 commit
  21. 18 Aug, 2014 1 commit
  22. 13 Aug, 2014 1 commit
  23. 22 Jul, 2014 1 commit
  24. 20 Mar, 2014 1 commit
    • antirez's avatar
      Added dictGetRandomKeys() to dict.c: mass get random entries. · 5317f5e9
      antirez authored
      This new function is useful to get a number of random entries from an
      hash table when we just need to do some sampling without particularly
      good distribution.
      
      It just jumps at a random place of the hash table and returns the first
      N items encountered by scanning linearly.
      
      The main usefulness of this function is to speedup Redis internal
      sampling of the key space, for example for key eviction or expiry.
      5317f5e9
  25. 10 Dec, 2013 1 commit
    • antirez's avatar
      dict.c: added optional callback to dictEmpty(). · 2eb781b3
      antirez authored
      Redis hash table implementation has many non-blocking features like
      incremental rehashing, however while deleting a large hash table there
      was no way to have a callback called to do some incremental work.
      
      This commit adds this support, as an optiona callback argument to
      dictEmpty() that is currently called at a fixed interval (one time every
      65k deletions).
      2eb781b3
  26. 25 Oct, 2013 1 commit
  27. 19 Aug, 2013 1 commit
    • antirez's avatar
      dict.c iterator API misuse protection. · 48cde3fe
      antirez authored
      dict.c allows the user to create unsafe iterators, that are iterators
      that will not touch the dictionary data structure in any way, preventing
      copy on write, but at the same time are limited in their usage.
      
      The limitation is that when itearting with an unsafe iterator, no call
      to other dictionary functions must be done inside the iteration loop,
      otherwise the dictionary may be incrementally rehashed resulting into
      missing elements in the set of the elements returned by the iterator.
      
      However after introducing this kind of iterators a number of bugs were
      found due to misuses of the API, and we are still finding
      bugs about this issue. The bugs are not trivial to track because the
      effect is just missing elements during the iteartion.
      
      This commit introduces auto-detection of the API misuse. The idea is
      that an unsafe iterator has a contract: from initialization to the
      release of the iterator the dictionary should not change.
      
      So we take a fingerprint of the dictionary state, xoring a few important
      dict properties when the unsafe iteartor is initialized. We later check
      when the iterator is released if the fingerprint is still the same. If it
      is not, we found a misuse of the iterator, as not allowed API calls
      changed the internal state of the dictionary.
      
      This code was checked against a real bug, issue #1240.
      
      This is what Redis prints (aborting) when a misuse is detected:
      
      Assertion failed: (iter->fingerprint == dictFingerprint(iter->d)),
      function dictReleaseIterator, file dict.c, line 587.
      48cde3fe
  28. 05 Oct, 2012 1 commit
    • antirez's avatar
      Hash function switched to murmurhash2. · da920e75
      antirez authored
      The previously used hash function, djbhash, is not secure against
      collision attacks even when the seed is randomized as there are simple
      ways to find seed-independent collisions.
      
      The new hash function appears to be safe (or much harder to exploit at
      least) in this case, and has better distribution.
      
      Better distribution does not always means that's better. For instance in
      a fast benchmark with "DEBUG POPULATE 1000000" I obtained the following
      results:
      
          1.6 seconds with djbhash
          2.0 seconds with murmurhash2
      
      This is due to the fact that djbhash will hash objects that follow the
      pattern `prefix:<id>` and where the id is numerically near, to near
      buckets. This improves the locality.
      
      However in other access patterns with keys that have no relation
      murmurhash2 has some (apparently minimal) speed advantage.
      
      On the other hand a better distribution should significantly
      improve the quality of the distribution of elements returned with
      dictGetRandomKey() that is used in SPOP, SRANDMEMBER, RANDOMKEY, and
      other commands.
      
      Everything considered, and under the suspect that this commit fixes a
      security issue in Redis, we are switching to the new hash function.
      If some serious speed regression will be found in the future we'll be able
      to step back easiliy.
      
      This commit fixes issue #663.
      da920e75
  29. 03 Oct, 2012 1 commit
  30. 21 Jan, 2012 1 commit
  31. 09 Nov, 2011 1 commit
  32. 08 Nov, 2011 4 commits
  33. 02 Nov, 2011 1 commit
  34. 10 May, 2011 1 commit