1. 18 Apr, 2017 20 commits
    • spinlock's avatar
    • antirez's avatar
      Cluster: discard pong times in the future. · d98ef35a
      antirez authored
      However we allow for 500 milliseconds of tolerance, in order to
      avoid often discarding semantically valid info (the node is up)
      because of natural few milliseconds desync among servers even when
      NTP is used.
      
      Note that anyway we should ping the node from time to time regardless and
      discover if it's actually down from our point of view, since no update
      is accepted while we have an active ping on the node.
      
      Related to #3929.
      d98ef35a
    • antirez's avatar
      Test: fix, hopefully, false PSYNC failure like in issue #2715. · e47c8e3f
      antirez authored
      And many other related Github issues... all reporting the same problem.
      There was probably just not enough backlog in certain unlucky runs.
      I'll ask people that can reporduce if they see now this as fixed as
      well.
      e47c8e3f
    • antirez's avatar
      Cluster: always add PFAIL nodes at end of gossip section. · 1e659a04
      antirez authored
      To rely on the fact that nodes in PFAIL state will be shared around by
      randomly adding them in the gossip section is a weak assumption,
      especially after changes related to sending less ping/pong packets.
      
      We want to always include gossip entries for all the nodes that are in
      PFAIL state, so that the PFAIL -> FAIL state promotion can happen much
      faster and reliably.
      
      Related to #3929.
      1e659a04
    • antirez's avatar
      Cluster: fix gossip section ping/pong times encoding. · 39d34487
      antirez authored
      The gossip section times are 32 bit, so cannot store the milliseconds
      time but just the seconds approximation, which is good enough for our
      uses. At the same time however, when comparing the gossip section times
      of other nodes with our node's view, we need to convert back to
      milliseconds.
      
      Related to #3929. Without this change the patch to reduce the traffic in
      the bus message does not work.
      39d34487
    • antirez's avatar
      78148d0e
    • antirez's avatar
      Cluster: decrease ping/pong traffic by trusting other nodes reports. · a5c1c77e
      antirez authored
      Cluster of bigger sizes tend to have a lot of traffic in the cluster bus
      just for failure detection: a node will try to get a ping reply from
      another node no longer than when the half the node timeout would elapsed,
      in order to avoid a false positive.
      
      However this means that if we have N nodes and the node timeout is set
      to, for instance M seconds, we'll have to ping N nodes every M/2
      seconds. This N*M/2 pings will receive the same number of pongs, so
      a total of N*M packets per node. However given that we have a total of N
      nodes doing this, the total number of messages will be N*N*M.
      
      In a 100 nodes cluster with a timeout of 60 seconds, this translates
      to a total of 100*100*30 packets per second, summing all the packets
      exchanged by all the nodes.
      
      This is, as you can guess, a lot... So this patch changes the
      implementation in a very simple way in order to trust the reports of
      other nodes: if a node A reports a node B as alive at least up to
      a given time, we update our view accordingly.
      
      The problem with this approach is that it could result into a subset of
      nodes being able to reach a given node X, and preventing others from
      detecting that is actually not reachable from the majority of nodes.
      So the above algorithm is refined by trusting other nodes only if we do
      not have currently a ping pending for the node X, and if there are no
      failure reports for that node.
      
      Since each node, anyway, pings 10 other nodes every second (one node
      every 100 milliseconds), anyway eventually even trusting the other nodes
      reports, we will detect if a given node is down from our POV.
      
      Now to understand the number of packets that the cluster would exchange
      for failure detection with the patch, we can start considering the
      random PINGs that the cluster sent anyway as base line:
      Each node sends 10 packets per second, so the total traffic if no
      additioal packets would be sent, including PONG packets, would be:
      
          Total messages per second = N*10*2
      
      However by trusting other nodes gossip sections will not AWALYS prevent
      pinging nodes for the "half timeout reached" rule all the times. The
      math involved in computing the actual rate as N and M change is quite
      complex and depends also on another parameter, which is the number of
      entries in the gossip section of PING and PONG packets. However it is
      possible to compare what happens in cluster of different sizes
      experimentally. After applying this patch a very important reduction in
      the number of packets exchanged is trivial to observe, without apparent
      impacts on the failure detection performances.
      
      Actual numbers with different cluster sizes should be published in the
      Reids Cluster documentation in the future.
      
      Related to #3929.
      a5c1c77e
    • antirez's avatar
      Cluster: collect more specific bus messages stats. · 51901396
      antirez authored
      First step in order to change Cluster in order to use less messages.
      Related to issue #3929.
      51901396
    • antirez's avatar
      f7b91b6c
    • antirez's avatar
      Set lua-time-limit default value at safe place. · 6e1489ae
      antirez authored
      Otherwise, as it was, it will overwrite whatever the user set.
      
      Close #3703.
      6e1489ae
    • antirez's avatar
      5fd841c0
    • antirez's avatar
      185b361a
    • lorneli's avatar
      Expire: Update comment of activeExpireCycle function · b740fc1e
      lorneli authored
      The macro REDIS_EXPIRELOOKUPS_TIME_PERC has been replaced by
      ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC in commit
      6500fabf.
      b740fc1e
    • antirez's avatar
      56cafcce
    • antirez's avatar
      Make more obvious why there was issue #3843. · a5b66da8
      antirez authored
      a5b66da8
    • antirez's avatar
      Fix modules blocking commands awake delay. · f60d6f09
      antirez authored
      If a thread unblocks a client blocked in a module command, by using the
      RedisMdoule_UnblockClient() API, the event loop may not be awaken until
      the next timeout of the multiplexing API or the next unrelated I/O
      operation on other clients. We actually want the client to be served
      ASAP, so a mechanism is needed in order for the unblocking API to inform
      Redis that there is a client to serve ASAP.
      
      This commit fixes the issue using the old trick of the pipe: when a
      client needs to be unblocked, a byte is written in a pipe. When we run
      the list of clients blocked in modules, we consume all the bytes
      written in the pipe. Writes and reads are performed inside the context
      of the mutex, so no race is possible in which we consume the bytes that
      are actually related to an awake request for a client that should still
      be put into the list of clients to unblock.
      
      It was verified that after the fix the server handles the blocked
      clients with the expected short delay.
      
      Thanks to @dvirsky for understanding there was such a problem and
      reporting it.
      f60d6f09
    • antirez's avatar
      Rax library updated. · c56668c8
      antirez authored
      c56668c8
    • antirez's avatar
      Cluster: hash slots tracking using a radix tree. · c4716d33
      antirez authored
      c4716d33
    • vienna's avatar
      a9fefbce
    • Dvir Volk's avatar
      17250409
  2. 22 Mar, 2017 8 commits
  3. 21 Feb, 2017 9 commits
    • antirez's avatar
      SipHash 2-4 -> SipHash 1-2. · 5a413303
      antirez authored
      For performance reasons we use a reduced rounds variant of
      SipHash. This should still provide enough protection and the
      effects in the hash table distribution are non existing.
      If some real world attack on SipHash 1-2 will be found we can
      trivially switch to something more secure. Anyway it is a
      big step forward from Murmurhash, for which it is trivial to
      generate *seed independent* colliding keys... The speed
      penatly introduced by SipHash 2-4, around 4%, was a too big
      price to pay compared to the effectiveness of the HashDoS
      attack against SipHash 1-2, and considering so far in the
      Redis history, no such an incident ever happened even while
      using trivially to collide hash functions.
      5a413303
    • antirez's avatar
      freeMemoryIfNeeded(): improve code and lazyfree handling. · a8cbc3ec
      antirez authored
      1. Refactor memory overhead computation into a function.
      2. Every 10 keys evicted, check if memory usage already reached
         the target value directly, since we otherwise don't count all
         the memory reclaimed by the background thread right now.
      a8cbc3ec
    • antirez's avatar
      857e6d56
    • antirez's avatar
      SipHash x86 optimizations. · 34387cea
      antirez authored
      34387cea
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · ba647598
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      ba647598
    • Salvatore Sanfilippo's avatar
      ARM: Avoid fast path for BITOP. · 2ee19d98
      Salvatore Sanfilippo authored
      GCC will produce certain unaligned multi load-store instructions
      that will be trapped by the Linux kernel since ARM v6 cannot
      handle them with unaligned addresses. Better to use the slower
      but safer implementation instead of generating the exception which
      should be anyway very slow.
      2ee19d98
    • Salvatore Sanfilippo's avatar
      ARM: Use libc malloc by default. · eb62cfea
      Salvatore Sanfilippo authored
      I'm not sure how much test Jemalloc gets on ARM, moreover
      compiling Redis with Jemalloc support in not very powerful
      devices, like most ARMs people will build Redis on, is extremely
      slow. It is possible to enable Jemalloc build anyway if needed
      by using "make MALLOC=jemalloc".
      eb62cfea
    • Salvatore Sanfilippo's avatar
      ARM: Avoid memcpy() in MurmurHash64A() if we are using 64 bit ARM. · 620e48b1
      Salvatore Sanfilippo authored
      However note that in architectures supporting 64 bit unaligned
      accesses memcpy(...,...,8) is likely translated to a simple
      word memory movement anyway.
      620e48b1
    • Salvatore Sanfilippo's avatar
  4. 20 Feb, 2017 2 commits
  5. 09 Feb, 2017 1 commit
    • antirez's avatar
      Fix MIGRATE closing of cached socket on error. · 8d55aeb5
      antirez authored
      After investigating issue #3796, it was discovered that MIGRATE
      could call migrateCloseSocket() after the original MIGRATE c->argv
      was already rewritten as a DEL operation. As a result the host/port
      passed to migrateCloseSocket() could be anything, often a NULL pointer
      that gets deferenced crashing the server.
      
      Now the socket is closed at an earlier time when there is a socket
      error in a later stage where no retry will be performed, before we
      rewrite the argument vector. Moreover a check was added so that later,
      in the socket_err label, there is no further attempt at closing the
      socket if the argument was rewritten.
      
      This fix should resolve the bug reported in #3796.
      8d55aeb5