1. 06 Dec, 2020 2 commits
    • Oran Agra's avatar
      Sanitize dump payload: fail RESTORE if memory allocation fails · 7ca00d69
      Oran Agra authored
      When RDB input attempts to make a huge memory allocation that fails,
      RESTORE should fail gracefully rather than die with panic
      7ca00d69
    • Wang Yuan's avatar
      Limit the main db and expires dictionaries to expand (#7954) · 75f9dec6
      Wang Yuan authored
      As we know, redis may reject user's requests or evict some keys if
      used memory is over maxmemory. Dictionaries expanding may make
      things worse, some big dictionaries, such as main db and expires dict,
      may eat huge memory at once for allocating a new big hash table and be
      far more than maxmemory after expanding.
      There are related issues: #4213 #4583
      
      More details, when expand dict in redis, we will allocate a new big
      ht[1] that generally is double of ht[0], The size of ht[1] will be
      very big if ht[0] already is big. For db dict, if we have more than
      64 million keys, we need to cost 1GB for ht[1] when dict expands.
      
      If the sum of used memory and new hash table of dict needed exceeds
      maxmemory, we shouldn't allow the dict to expand. Because, if we
      enable keys eviction, we still couldn't add much more keys after
      eviction and rehashing, what's worse, redis will keep less keys when
      redis only remains a little memory for storing new hash table instead
      of users' data. Moreover users can't write data in redis if disable
      keys eviction.
      
      What this commit changed ?
      
      Add a new member function expandAllowed for dict type, it provide a way
      for caller to allow expand or not. We expose two parameters for this
      function: more memory needed for expanding and dict current load factor,
      users can implement a function to make a decision by them.
      For main db dict and expires dict type, these dictionaries may be very
      big and cost huge memory for expanding, so we implement a judgement
      function: we can stop dict to expand provisionally if used memory will
      be over maxmemory after dict expands, but to guarantee the performance
      of redis, we still allow dict to expand if dict load factor exceeds the
      safe load factor.
      Add test cases to verify we don't allow main db to expand when left
      memory is not enough, so that avoid keys eviction.
      
      Other changes:
      
      For new hash table size when expand. Before this commit, the size is
      that double used of dict and later _dictNextPower. Actually we aim to
      control a dict load factor between 0.5 and 1.0. Now we replace *2 with
      +1, since the first check is that used >= size, the outcome of before
      will usually be the same as _dictNextPower(used+1). The only case where
      it'll differ is when dict_can_resize is false during fork, so that later
      the _dictNextPower(used*2) will cause the dict to jump to *4 (i.e.
      _dictNextPower(1025*2) will return 4096).
      Fix rehash test cases due to changing algorithm of new hash table size
      when expand.
      75f9dec6
  2. 18 Feb, 2019 1 commit
  3. 05 Dec, 2017 1 commit
  4. 20 Feb, 2017 1 commit
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · adeed29a
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      adeed29a
  5. 02 Jan, 2017 1 commit
  6. 30 Dec, 2016 1 commit
  7. 14 Sep, 2016 2 commits
    • antirez's avatar
      dict.c: dictReplaceRaw() -> dictAddOrFind(). · 09a50d34
      antirez authored
      What they say about "naming things" in programming?
      09a50d34
    • oranagra's avatar
      dict.c: introduce dictUnlink(). · afcbcc0e
      oranagra authored
      Notes by @antirez:
      
      This patch was picked from a larger commit by Oran and adapted to change
      the API a bit. The basic idea is to avoid double lookups when there is
      to use the value of the deleted entry.
      
      BEFORE:
      
          entry = dictFind( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictDelete(...);         /* 2nd lookup. */
      
      AFTER:
      
          entry = dictUnlink( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictFreeUnlinkedEntry(entry); /* No lookups!. */
      afcbcc0e
  8. 12 Sep, 2016 1 commit
  9. 01 Oct, 2015 1 commit
  10. 14 Jul, 2015 1 commit
    • antirez's avatar
      DEBUG HTSTATS <dbid> added. · 0f64080d
      antirez authored
      The command reports information about the hash table internal state
      representing the specified database ID.
      
      This can be used in order to investigate rehashings, memory usage issues
      and for other debugging purposes.
      0f64080d
  11. 11 Feb, 2015 2 commits
  12. 26 Aug, 2014 1 commit
  13. 18 Aug, 2014 1 commit
  14. 13 Aug, 2014 1 commit
  15. 22 Jul, 2014 1 commit
  16. 20 Mar, 2014 1 commit
    • antirez's avatar
      Added dictGetRandomKeys() to dict.c: mass get random entries. · 5317f5e9
      antirez authored
      This new function is useful to get a number of random entries from an
      hash table when we just need to do some sampling without particularly
      good distribution.
      
      It just jumps at a random place of the hash table and returns the first
      N items encountered by scanning linearly.
      
      The main usefulness of this function is to speedup Redis internal
      sampling of the key space, for example for key eviction or expiry.
      5317f5e9
  17. 10 Dec, 2013 1 commit
    • antirez's avatar
      dict.c: added optional callback to dictEmpty(). · 2eb781b3
      antirez authored
      Redis hash table implementation has many non-blocking features like
      incremental rehashing, however while deleting a large hash table there
      was no way to have a callback called to do some incremental work.
      
      This commit adds this support, as an optiona callback argument to
      dictEmpty() that is currently called at a fixed interval (one time every
      65k deletions).
      2eb781b3
  18. 25 Oct, 2013 1 commit
  19. 19 Aug, 2013 1 commit
    • antirez's avatar
      dict.c iterator API misuse protection. · 48cde3fe
      antirez authored
      dict.c allows the user to create unsafe iterators, that are iterators
      that will not touch the dictionary data structure in any way, preventing
      copy on write, but at the same time are limited in their usage.
      
      The limitation is that when itearting with an unsafe iterator, no call
      to other dictionary functions must be done inside the iteration loop,
      otherwise the dictionary may be incrementally rehashed resulting into
      missing elements in the set of the elements returned by the iterator.
      
      However after introducing this kind of iterators a number of bugs were
      found due to misuses of the API, and we are still finding
      bugs about this issue. The bugs are not trivial to track because the
      effect is just missing elements during the iteartion.
      
      This commit introduces auto-detection of the API misuse. The idea is
      that an unsafe iterator has a contract: from initialization to the
      release of the iterator the dictionary should not change.
      
      So we take a fingerprint of the dictionary state, xoring a few important
      dict properties when the unsafe iteartor is initialized. We later check
      when the iterator is released if the fingerprint is still the same. If it
      is not, we found a misuse of the iterator, as not allowed API calls
      changed the internal state of the dictionary.
      
      This code was checked against a real bug, issue #1240.
      
      This is what Redis prints (aborting) when a misuse is detected:
      
      Assertion failed: (iter->fingerprint == dictFingerprint(iter->d)),
      function dictReleaseIterator, file dict.c, line 587.
      48cde3fe
  20. 05 Oct, 2012 1 commit
    • antirez's avatar
      Hash function switched to murmurhash2. · da920e75
      antirez authored
      The previously used hash function, djbhash, is not secure against
      collision attacks even when the seed is randomized as there are simple
      ways to find seed-independent collisions.
      
      The new hash function appears to be safe (or much harder to exploit at
      least) in this case, and has better distribution.
      
      Better distribution does not always means that's better. For instance in
      a fast benchmark with "DEBUG POPULATE 1000000" I obtained the following
      results:
      
          1.6 seconds with djbhash
          2.0 seconds with murmurhash2
      
      This is due to the fact that djbhash will hash objects that follow the
      pattern `prefix:<id>` and where the id is numerically near, to near
      buckets. This improves the locality.
      
      However in other access patterns with keys that have no relation
      murmurhash2 has some (apparently minimal) speed advantage.
      
      On the other hand a better distribution should significantly
      improve the quality of the distribution of elements returned with
      dictGetRandomKey() that is used in SPOP, SRANDMEMBER, RANDOMKEY, and
      other commands.
      
      Everything considered, and under the suspect that this commit fixes a
      security issue in Redis, we are switching to the new hash function.
      If some serious speed regression will be found in the future we'll be able
      to step back easiliy.
      
      This commit fixes issue #663.
      da920e75
  21. 03 Oct, 2012 1 commit
  22. 21 Jan, 2012 1 commit
  23. 09 Nov, 2011 1 commit
  24. 08 Nov, 2011 4 commits
  25. 02 Nov, 2011 1 commit
  26. 10 May, 2011 1 commit
  27. 03 Nov, 2010 1 commit
  28. 01 Jul, 2010 1 commit
    • antirez's avatar
      redis.c split into many different C files. · e2641e09
      antirez authored
      networking related stuff moved into networking.c
      
      moved more code
      
      more work on layout of source code
      
      SDS instantaneuos memory saving. By Pieter and Salvatore at VMware ;)
      
      cleanly compiling again after the first split, now splitting it in more C files
      
      moving more things around... work in progress
      
      split replication code
      
      splitting more
      
      Sets split
      
      Hash split
      
      replication split
      
      even more splitting
      
      more splitting
      
      minor change
      e2641e09
  29. 16 Apr, 2010 1 commit
  30. 15 Apr, 2010 2 commits
  31. 08 Apr, 2010 1 commit
  32. 19 Feb, 2010 1 commit
  33. 08 Jun, 2009 1 commit