1. 06 Mar, 2017 1 commit
  2. 20 Feb, 2017 1 commit
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · adeed29a
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      adeed29a
  3. 12 Jan, 2017 1 commit
  4. 14 Dec, 2016 1 commit
  5. 13 Dec, 2016 1 commit
    • antirez's avatar
      Replication: fix the infamous key leakage of writable slaves + EXPIRE. · 04542cff
      antirez authored
      BACKGROUND AND USE CASEj
      
      Redis slaves are normally write only, however the supprot a "writable"
      mode which is very handy when scaling reads on slaves, that actually
      need write operations in order to access data. For instance imagine
      having slaves replicating certain Sets keys from the master. When
      accessing the data on the slave, we want to peform intersections between
      such Sets values. However we don't want to intersect each time: to cache
      the intersection for some time often is a good idea.
      
      To do so, it is possible to setup a slave as a writable slave, and
      perform the intersection on the slave side, perhaps setting a TTL on the
      resulting key so that it will expire after some time.
      
      THE BUG
      
      Problem: in order to have a consistent replication, expiring of keys in
      Redis replication is up to the master, that synthesize DEL operations to
      send in the replication stream. However slaves logically expire keys
      by hiding them from read attempts from clients so that if the master did
      not promptly sent a DEL, the client still see logically expired keys
      as non existing.
      
      Because slaves don't actively expire keys by actually evicting them but
      just masking from the POV of read operations, if a key is created in a
      writable slave, and an expire is set, the key will be leaked forever:
      
      1. No DEL will be received from the master, which does not know about
      such a key at all.
      
      2. No eviction will be performed by the slave, since it needs to disable
      eviction because it's up to masters, otherwise consistency of data is
      lost.
      
      THE FIX
      
      In order to fix the problem, the slave should be able to tag keys that
      were created in the slave side and have an expire set in some way.
      
      My solution involved using an unique additional dictionary created by
      the writable slave only if needed. The dictionary is obviously keyed by
      the key name that we need to track: all the keys that are set with an
      expire directly by a client writing to the slave are tracked.
      
      The value in the dictionary is a bitmap of all the DBs where such a key
      name need to be tracked, so that we can use a single dictionary to track
      keys in all the DBs used by the slave (actually this limits the solution
      to the first 64 DBs, but the default with Redis is to use 16 DBs).
      
      This solution allows to pay both a small complexity and CPU penalty,
      which is zero when the feature is not used, actually. The slave-side
      eviction is encapsulated in code which is not coupled with the rest of
      the Redis core, if not for the hook to track the keys.
      
      TODO
      
      I'm doing the first smoke tests to see if the feature works as expected:
      so far so good. Unit tests should be added before merging into the
      4.0 branch.
      04542cff
  6. 30 Nov, 2016 2 commits
  7. 24 Nov, 2016 1 commit
  8. 31 Oct, 2016 1 commit
  9. 13 Oct, 2016 2 commits
  10. 07 Oct, 2016 4 commits
  11. 06 Oct, 2016 2 commits
    • antirez's avatar
      Module: Ability to get context from IO context. · 152c1b68
      antirez authored
      It was noted by @dvirsky that it is not possible to use string functions
      when writing the AOF file. This sometimes is critical since the command
      rewriting may need to be built in the context of the AOF callback, and
      without access to the context, and the limited types that the AOF
      production functions will accept, this can be an issue.
      
      Moreover there are other needs that we can't anticipate regarding the
      ability to use Redis Modules APIs using the context in order to build
      representations to emit AOF / RDB.
      
      Because of this a new API was added that allows the user to get a
      temporary context from the IO context. The context is auto released
      if obtained when the RDB / AOF callback returns.
      
      Calling multiple time the function to get the context, always returns
      the same one, since it is invalid to have more than a single context.
      152c1b68
    • antirez's avatar
      Copyright notice added to module.c. · 72279e3e
      antirez authored
      72279e3e
  12. 02 Oct, 2016 2 commits
  13. 21 Sep, 2016 1 commit
  14. 14 Sep, 2016 1 commit
    • oranagra's avatar
      dict.c: introduce dictUnlink(). · afcbcc0e
      oranagra authored
      Notes by @antirez:
      
      This patch was picked from a larger commit by Oran and adapted to change
      the API a bit. The basic idea is to avoid double lookups when there is
      to use the value of the deleted entry.
      
      BEFORE:
      
          entry = dictFind( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictDelete(...);         /* 2nd lookup. */
      
      AFTER:
      
          entry = dictUnlink( ... ); /* 1st lookup. */
          /* Do somethjing with the entry. */
          dictFreeUnlinkedEntry(entry); /* No lookups!. */
      afcbcc0e
  15. 09 Sep, 2016 1 commit
  16. 03 Aug, 2016 2 commits
  17. 02 Aug, 2016 1 commit
    • antirez's avatar
      Modules: StringAppendBuffer() and ability to retain strings. · 7829e4ed
      antirez authored
      RedisModule_StringRetain() allows, when automatic memory management is
      on, to keep string objects living after the callback returns. Can also
      be used in order to use Redis reference counting of objects inside
      modules.
      
      The reason why this is useful is that sometimes when implementing new
      data types we want to reference RedisModuleString objects inside the
      module private data structures, so those string objects must be valid
      after the callback returns even if not referenced inside the Redis key
      space.
      7829e4ed
  18. 23 Jun, 2016 6 commits
    • antirez's avatar
      Actually remove static from #3331. · 0f484d83
      antirez authored
      I forgot -a when amending in the previous commit.
      0f484d83
    • antirez's avatar
      Minor change to conform PR #3331 to Redis code base style. · c0ca87dc
      antirez authored
      Also avoid "static" in order to have symbols during crashes.
      c0ca87dc
    • antirez's avatar
      Modules: changes to logging function. · 4b12c6a3
      antirez authored
      This commit changes what provided by PR #3315 (merged) in order to
      let the user specify the log level as a string.
      
      The define could be also used, but when this happens, they must be
      decoupled from the defines in the Redis core, like in the other part of
      the Redis modules implementations, so that a switch statement (or a
      function) remaps between the two, otherwise we are no longer free to
      change the internal Redis defines.
      4b12c6a3
    • Yossi Gottlieb's avatar
      715794b8
    • antirez's avatar
      Commit change in autoMemoryFreed(): first -> last. · b5072897
      antirez authored
      It's more natural to call the last entry added as "last", the original
      commet got me confused until I actually read the code.
      b5072897
    • antirez's avatar
      Modules: implement zig-zag scanning in autoMemoryFreed(). · f2dbc02f
      antirez authored
      Most of the time to check the last element is the way to go, however
      there are patterns where the contrary is the best choice. Zig-zag
      scanning implemented in this commmit always checks the obvious element
      first (the last added -- think at a loop where the last element
      allocated gets freed again and again), and continues checking one
      element in the head and one in the tail.
      
      Thanks to @dvisrky that fixed the original implementation of the
      function and proposed zig zag scanning.
      f2dbc02f
  19. 22 Jun, 2016 3 commits
  20. 21 Jun, 2016 1 commit
  21. 20 Jun, 2016 1 commit
  22. 13 Jun, 2016 2 commits
  23. 05 Jun, 2016 2 commits