1. 23 Jul, 2018 1 commit
    • Oran Agra's avatar
      fix rare replication stream corruption with disk-based replication · 9535c215
      Oran Agra authored
      The slave sends \n keepalive messages to the master while parsing the rdb,
      and later sends REPLCONF ACK once a second. rarely, the master recives both
      a linefeed char and a REPLCONF in the same read, \n*3\r\n$8\r\nREPLCONF\r\n...
      and it tries to trim two chars (\r\n) from the query buffer,
      trimming the '*' from *3\r\n$8\r\nREPLCONF\r\n...
      
      then the master tries to process a command starting with '3' and replies to
      the slave a bunch of -ERR and one +OK.
      although the slave silently ignores these (prints a log message), this corrupts
      the replication offset at the slave since the slave increases the replication
      offset, and the master did not.
      
      other than the fix in processInlineBuffer, i did several other improvments
      while hunting this very rare bug.
      
      - when redis replies with "unknown command" it includes a portion of the
        arguments, not just the command name. so it would be easier to understand
        what was recived, in my case, on the slave side,  it was -ERR, but
        the "arguments" were the interesting part (containing info on the error).
      - about a year ago i added code in addReplyErrorLength to print the error to
        the log in case of a reply to master (since this string isn't actually
        trasmitted to the master), now changed that block to print a similar log
        message to indicate an error being sent from the master to the slave.
        note that the slave is marked as CLIENT_SLAVE only after PSYNC was received,
        so this will not cause any harm for REPLCONF, and will only indicate problems
        that are gonna corrupt the replication stream anyway.
      - two places were c->reply was emptied, and i wanted to reset sentlen
        this is a precaution (i did not actually see such a problem), since a
        non-zero sentlen will cause corruption to be transmitted on the socket.
      9535c215
  2. 14 Mar, 2018 1 commit
    • antirez's avatar
      Cluster: ability to prevent slaves from failing over their masters. · 70597a30
      antirez authored
      This commit, in some parts derived from PR #3041 which is no longer
      possible to merge (because the user deleted the original branch),
      implements the ability of slaves to have a special configuration
      preventing that they try to start a failover when the master is failing.
      
      There are multiple reasons for wanting this, and the feautre was
      requested in issue #3021 time ago.
      
      The differences between this patch and the original PR are the
      following:
      
      1. The flag is saved/loaded on the nodes configuration.
      2. The 'myself' node is now flag-aware, the flag is updated as needed
         when the configuration is changed via CONFIG SET.
      3. The flag name uses NOFAILOVER instead of NO_FAILOVER to be consistent
         with existing NOADDR.
      4. The redis.conf documentation was rewritten.
      
      Thanks to @deep011 for the original patch.
      70597a30
  3. 19 Feb, 2018 1 commit
    • antirez's avatar
      Track number of logically expired keys still in memory. · 83923afa
      antirez authored
      This commit adds two new fields in the INFO output, stats section:
      
      expired_stale_perc:0.34
      expired_time_cap_reached_count:58
      
      The first field is an estimate of the number of keys that are yet in
      memory but are already logically expired. They reason why those keys are
      yet not reclaimed is because the active expire cycle can't spend more
      time on the process of reclaiming the keys, and at the same time nobody
      is accessing such keys. However as the active expire cycle runs, while
      it will eventually have to return to the caller, because of time limit
      or because there are less than 25% of keys logically expired in each
      given database, it collects the stats in order to populate this INFO
      field.
      
      Note that expired_stale_perc is a running average, where the current
      sample accounts for 5% and the history for 95%, so you'll see it
      changing smoothly over time.
      
      The other field, expired_time_cap_reached_count, counts the number
      of times the expire cycle had to stop, even if still it was finding a
      sizeable number of keys yet to expire, because of the time limit.
      This allows people handling operations to understand if the Redis
      server, during mass-expiration events, is able to collect keys fast
      enough usually. It is normal for this field to increment during mass
      expires, but normally it should very rarely increment. When instead it
      constantly increments, it means that the current workloads is using
      a very important percentage of CPU time to expire keys.
      
      This feature was created thanks to the hints of Rashmi Ramesh and
      Bart Robinson from Twitter. In private email exchanges, they noted how
      it was important to improve the observability of this parameter in the
      Redis server. Actually in big deployments, the amount of keys that are
      yet to expire in each server, even if they are logically expired, may
      account for a very big amount of wasted memory.
      83923afa
  4. 24 Jan, 2018 1 commit
  5. 18 Jan, 2018 5 commits
  6. 30 Nov, 2017 2 commits
    • antirez's avatar
      Prevent corruption of server.executable after DEBUG RESTART. · ab3d3aca
      antirez authored
      Doing the following ended with a broken server.executable:
      
      1. Start Redis with src/redis-server
      2. Send CONFIG SET DIR /tmp/
      3. Send DEBUG RESTART
      
      At this point we called execve with an argv[0] that is no longer related
      to the new path. So after the restart the absolute path of the
      executable is recomputed in the wrong way. With this fix we pass the
      absolute path already computed as argv[0].
      ab3d3aca
    • antirez's avatar
      Be more verbose when DEBUG RESTART fails. · b7c7edf9
      antirez authored
      b7c7edf9
  7. 28 Nov, 2017 1 commit
  8. 24 Nov, 2017 1 commit
  9. 23 Nov, 2017 1 commit
  10. 21 Nov, 2017 1 commit
  11. 20 Sep, 2017 1 commit
    • antirez's avatar
      PSYNC2: Fix the way replication info is saved/loaded from RDB. · 097a5556
      antirez authored
      This commit attempts to fix a number of bugs reported in #4316.
      They are related to the way replication info like replication ID,
      offsets, and currently selected DB in the master client, are stored
      and loaded by Redis. In order to avoid inconsistencies the changes in
      this commit try to enforce that:
      
      1. Replication information are only stored when the RDB file is
      generated by a slave that has a valid 'master' client, so that we can
      always extract the currently selected DB.
      2. When replication informations are persisted in the RDB file, all the
      info for a successful PSYNC or nothing is persisted.
      3. The RDB replication informations are only loaded if the instance is
      configured as a slave, otherwise a master can start with IDs that relate
      to a different history of the data set, and stil retain such IDs in the
      future while receiving unrelated writes.
      097a5556
  12. 18 Sep, 2017 1 commit
    • Oran Agra's avatar
      Flush append only buffers before existing. · 8651e5d5
      Oran Agra authored
      when SHUTDOWN command is recived it is possible that some of the recent
      command were not yet flushed from the AOF buffer, and the server
      experiences data loss at shutdown.
      8651e5d5
  13. 02 Aug, 2017 1 commit
  14. 14 Jul, 2017 1 commit
  15. 06 Jul, 2017 2 commits
  16. 30 Jun, 2017 2 commits
    • antirez's avatar
      Added GEORADIUS(BYMEMBER)_RO variants for read-only operations. · bdd6de96
      antirez authored
      Issue #4084 shows how for a design error, GEORADIUS is a write command
      because of the STORE option. Because of this it does not work
      on readonly slaves, gets redirected to masters in Redis Cluster even
      when the connection is in READONLY mode and so forth.
      
      To break backward compatibility at this stage, with Redis 4.0 to be in
      advanced RC state, is problematic for the user base. The API can be
      fixed into the unstable branch soon if we'll decide to do so in order to
      be more consistent, and reease Redis 5.0 with this incompatibility in
      the future. This is still unclear.
      
      However, the ability to scale GEO queries in slaves easily is too
      important so this commit adds two read-only variants to the GEORADIUS
      and GEORADIUSBYMEMBER command: GEORADIUS_RO and GEORADIUSBYMEMBER_RO.
      The commands are exactly as the original commands, but they do not
      accept the STORE and STOREDIST options.
      bdd6de96
    • Suraj Narkhede's avatar
  17. 22 Jun, 2017 1 commit
  18. 15 Jun, 2017 1 commit
  19. 14 Jun, 2017 1 commit
  20. 19 May, 2017 1 commit
  21. 11 May, 2017 7 commits
  22. 28 Apr, 2017 1 commit
    • antirez's avatar
      Check event loop creation return value. Fix #3951. · b506eb74
      antirez authored
      Normally we never check for OOM conditions inside Redis since the
      allocator will always return a pointer or abort the program on OOM
      conditons. However we cannot have control on epool_create(), that may
      fail for kernel OOM (according to the manual page) even if all the
      parameters are correct, so the function aeCreateEventLoop() may indeed
      return NULL and this condition must be checked.
      b506eb74
  23. 18 Apr, 2017 2 commits
    • antirez's avatar
      Set lua-time-limit default value at safe place. · 6e1489ae
      antirez authored
      Otherwise, as it was, it will overwrite whatever the user set.
      
      Close #3703.
      6e1489ae
    • antirez's avatar
      Fix modules blocking commands awake delay. · f60d6f09
      antirez authored
      If a thread unblocks a client blocked in a module command, by using the
      RedisMdoule_UnblockClient() API, the event loop may not be awaken until
      the next timeout of the multiplexing API or the next unrelated I/O
      operation on other clients. We actually want the client to be served
      ASAP, so a mechanism is needed in order for the unblocking API to inform
      Redis that there is a client to serve ASAP.
      
      This commit fixes the issue using the old trick of the pipe: when a
      client needs to be unblocked, a byte is written in a pipe. When we run
      the list of clients blocked in modules, we consume all the bytes
      written in the pipe. Writes and reads are performed inside the context
      of the mutex, so no race is possible in which we consume the bytes that
      are actually related to an awake request for a client that should still
      be put into the list of clients to unblock.
      
      It was verified that after the fix the server handles the blocked
      clients with the expected short delay.
      
      Thanks to @dvirsky for understanding there was such a problem and
      reporting it.
      f60d6f09
  24. 21 Feb, 2017 1 commit
    • antirez's avatar
      Use SipHash hash function to mitigate HashDos attempts. · ba647598
      antirez authored
      This change attempts to switch to an hash function which mitigates
      the effects of the HashDoS attack (denial of service attack trying
      to force data structures to worst case behavior) while at the same time
      providing Redis with an hash function that does not expect the input
      data to be word aligned, a condition no longer true now that sds.c
      strings have a varialbe length header.
      
      Note that it is possible sometimes that even using an hash function
      for which collisions cannot be generated without knowing the seed,
      special implementation details or the exposure of the seed in an
      indirect way (for example the ability to add elements to a Set and
      check the return in which Redis returns them with SMEMBERS) may
      make the attacker's life simpler in the process of trying to guess
      the correct seed, however the next step would be to switch to a
      log(N) data structure when too many items in a single bucket are
      detected: this seems like an overkill in the case of Redis.
      
      SPEED REGRESION TESTS:
      
      In order to verify that switching from MurmurHash to SipHash had
      no impact on speed, a set of benchmarks involving fast insertion
      of 5 million of keys were performed.
      
      The result shows Redis with SipHash in high pipelining conditions
      to be about 4% slower compared to using the previous hash function.
      However this could partially be related to the fact that the current
      implementation does not attempt to hash whole words at a time but
      reads single bytes, in order to have an output which is endian-netural
      and at the same time working on systems where unaligned memory accesses
      are a problem.
      
      Further X86 specific optimizations should be tested, the function
      may easily get at the same level of MurMurHash2 if a few optimizations
      are performed.
      ba647598
  25. 12 Jan, 2017 1 commit
  26. 20 Dec, 2016 1 commit