1. 22 Feb, 2013 2 commits
    • antirez's avatar
      Cluster: new state information, cluster size. · d218a4e2
      antirez authored
      The definition of cluster size is: the number of known nodes in the
      cluster that are masters and serving at least an hash slot.
      d218a4e2
    • antirez's avatar
      Cluster: introduced a failure reports system. · 97492977
      antirez authored
      A §Redis Cluster node used to mark a node as failing when itself
      detected a failure for that node, and a single acknowledge was received
      about the possible failure state.
      
      The new API will be used in order to possible to require that N other
      nodes have a PFAIL or FAIL state for a given node for a node to set it
      as failing.
      97492977
  2. 20 Feb, 2013 1 commit
  3. 15 Feb, 2013 1 commit
  4. 14 Feb, 2013 3 commits
  5. 12 Feb, 2013 5 commits
    • antirez's avatar
      dc24a6b1
    • antirez's avatar
    • antirez's avatar
      07888202
    • antirez's avatar
      Use replicationFeedSlaves() to send PING to slaves. · 4b83ad4e
      antirez authored
      A Redis master sends PING commands to slaves from time to time: doing
      this ensures that even if absence of writes, the master->slave channel
      remains active and the slave can feel the master presence, instead of
      closing the connection for timeout.
      
      This commit changes the way PINGs are sent to slaves in order to use the
      standard interface used to replicate all the other commands, that is,
      the function replicationFeedSlaves().
      
      With this change the stream of commands sent to every slave is exactly
      the same regardless of their exact state (Transferring RDB for first
      synchronization or slave already online). With the previous
      implementation the PING was only sent to online slaves, with the result
      that the output stream from master to slaves was not identical for all
      the slaves: this is a problem if we want to implement partial resyncs in
      the future using a global replication stream offset.
      
      TL;DR: this commit should not change the behaviour in practical terms,
      but is just something in preparation for partial resynchronization
      support.
      4b83ad4e
    • antirez's avatar
      Emit SELECT to slaves in a centralized way. · 7465ac7a
      antirez authored
      Before this commit every Redis slave had its own selected database ID
      state. This was not actually useful as the emitted stream of commands
      is identical for all the slaves.
      
      Now the the currently selected database is a global state that is set to
      -1 when a new slave is attached, in order to force the SELECT command to
      be re-emitted for all the slaves.
      
      This change is useful in order to implement replication partial
      resynchronization in the future, as makes sure that the stream of
      commands received by slaves, including SELECT commands, are exactly the
      same for every slave connected, at any time.
      
      In this way we could have a global offset that can identify a specific
      piece of the master -> slaves stream of commands.
      7465ac7a
  6. 08 Feb, 2013 1 commit
  7. 05 Feb, 2013 2 commits
    • antirez's avatar
      b70b459b
    • charsyam's avatar
      Turn off TCP_NODELAY on the slave socket after SYNC. · c85647f3
      charsyam authored
      Further details from @antirez:
      
      It was reported by @StopForumSpam on Twitter that the Redis replication
      link was strangely using multiple TCP packets for multiple commands.
      This wastes a lot of bandwidth and is due to the TCP_NODELAY option we
      enable on the socket after accepting a new connection.
      
      However the master -> slave channel is a one-way channel since Redis
      replication is asynchronous, so there is no point in trying to reduce
      the latency, we should aim to reduce the bandwidth. For this reason this
      commit introduces the ability to disable the nagle algorithm on the
      socket after a successful SYNC.
      
      This feature is off by default because the delay can be up to 40
      milliseconds with normally configured Linux kernels.
      c85647f3
  8. 28 Jan, 2013 3 commits
    • antirez's avatar
      Keyspace events: it is now possible to select subclasses of events. · fce016d3
      antirez authored
      When keyspace events are enabled, the overhead is not sever but
      noticeable, so this commit introduces the ability to select subclasses
      of events in order to avoid to generate events the user is not
      interested in.
      
      The events can be selected using redis.conf or CONFIG SET / GET.
      fce016d3
    • antirez's avatar
      Fix decrRefCount() prototype from void to robj pointer. · 8766e810
      antirez authored
      decrRefCount used to get its argument as a void* pointer in order to be
      used as destructor where a 'void free_object(void*)' prototype is
      expected. However this made simpler to introduce bugs by freeing the
      wrong pointer. This commit fixes the argument type and introduces a new
      wrapper called decrRefCountVoid() that can be used when the void*
      argument is needed.
      8766e810
    • antirez's avatar
      Keyspace events notification API. · 4cdbce34
      antirez authored
      4cdbce34
  9. 19 Jan, 2013 1 commit
  10. 15 Jan, 2013 1 commit
    • antirez's avatar
      CLIENT GETNAME and CLIENT SETNAME introduced. · 1971740f
      antirez authored
      Sometimes it is much simpler to debug complex Redis installations if it
      is possible to assign clients a name that is displayed in the CLIENT
      LIST output.
      
      This is the case, for example, for "leaked" connections. The ability to
      provide a name to the client makes it quite trivial to understand what
      is the part of the code implementing the client not releasing the
      resources appropriately.
      
      Behavior:
      
          CLIENT SETNAME: set a name for the client, or remove the current
                          name if an empty name is set.
          CLIENT GETNAME: get the current name, or a nil.
          CLIENT LIST: now displays the client name if any.
      
      Thanks to Mark Gravell for pushing this idea forward.
      1971740f
  11. 14 Dec, 2012 1 commit
    • antirez's avatar
      serverCron() frequency is now a runtime parameter (was REDIS_HZ). · f1481d4a
      antirez authored
      REDIS_HZ is the frequency our serverCron() function is called with.
      A more frequent call to this function results into less latency when the
      server is trying to handle very expansive background operations like
      mass expires of a lot of keys at the same time.
      
      Redis 2.4 used to have an HZ of 10. This was good enough with almost
      every setup, but the incremental key expiration algorithm was working a
      bit better under *extreme* pressure when HZ was set to 100 for Redis
      2.6.
      
      However for most users a latency spike of 30 milliseconds when million
      of keys are expiring at the same time is acceptable, on the other hand a
      default HZ of 100 in Redis 2.6 was causing idle instances to use some
      CPU time compared to Redis 2.4. The CPU usage was in the order of 0.3%
      for an idle instance, however this is a shame as more energy is consumed
      by the server, if not important resources.
      
      This commit introduces HZ as a runtime parameter, that can be queried by
      INFO or CONFIG GET, and can be modified with CONFIG SET. At the same
      time the default frequency is set back to 10.
      
      In this way we default to a sane value of 10, but allows users to
      easily switch to values up to 500 for near real-time applications if
      needed and if they are willing to pay this small CPU usage penalty.
      f1481d4a
  12. 02 Dec, 2012 1 commit
    • antirez's avatar
      Blocking POP: use a dictionary to store keys clinet side. · 2f87cf8b
      antirez authored
      To store the keys we block for during a blocking pop operation, in the
      case the client is blocked for more data to arrive, we used a simple
      linear array of redis objects, in the blockingState structure:
      
          robj **keys;
          int count;
      
      However in order to fix issue #801 we also use a dictionary in order to
      avoid to end in the blocked clients queue for the same key multiple
      times with the same client.
      
      The dictionary was only temporary, just to avoid duplicates, but since
      we create / destroy it there is no point in doing this duplicated work,
      so this commit simply use a dictionary as the main structure to store
      the keys we are blocked for. So instead of the previous fields we now
      just have:
      
          dict *keys;
      
      This simplifies the code and reduces the work done by the server during
      a blocking POP operation.
      2f87cf8b
  13. 29 Nov, 2012 1 commit
    • antirez's avatar
      Introduced the Build ID in INFO and --version output. · 2f62c966
      antirez authored
      The idea is to be able to identify a build in a unique way, so for
      instance after a bug report we can recognize that the build is the one
      of a popular Linux distribution and perform the debugging in the same
      environment.
      2f62c966
  14. 22 Nov, 2012 2 commits
    • antirez's avatar
      EVALSHA is now case insensitive. · 95f68f7b
      antirez authored
      EVALSHA used to crash if the SHA1 was not lowercase (Issue #783).
      Fixed using a case insensitive dictionary type for the sha -> script
      map used for replication of scripts.
      95f68f7b
    • antirez's avatar
      Safer handling of MULTI/EXEC on errors. · 3d139127
      antirez authored
      After the transcation starts with a MULIT, the previous behavior was to
      return an error on problems such as maxmemory limit reached. But still
      to execute the transaction with the subset of queued commands on EXEC.
      
      While it is true that the client was able to check for errors
      distinguish QUEUED by an error reply, MULTI/EXEC in most client
      implementations uses pipelining for speed, so all the commands and EXEC
      are sent without caring about replies.
      
      With this change:
      
      1) EXEC fails if at least one command was not queued because of an
      error. The EXECABORT error is used.
      2) A generic error is always reported on EXEC.
      3) The client DISCARDs the MULTI state after a failed EXEC, otherwise
      pipelining multiple transactions would be basically impossible:
      After a failed EXEC the next transaction would be simply queued as
      the tail of the previous transaction.
      3d139127
  15. 11 Nov, 2012 1 commit
    • antirez's avatar
      MIGRATE TCP connections caching. · e23d281e
      antirez authored
      By caching TCP connections used by MIGRATE to chat with other Redis
      instances a 5x performance improvement was measured with
      redis-benchmark against small keys.
      
      This can dramatically speedup cluster resharding and other processes
      where an high load of MIGRATE commands are used.
      e23d281e
  16. 08 Nov, 2012 1 commit
  17. 01 Nov, 2012 1 commit
  18. 31 Oct, 2012 1 commit
  19. 16 Oct, 2012 1 commit
    • antirez's avatar
      Fix MULTI / EXEC rendering in MONITOR output. · a1b1c1ea
      antirez authored
      Before of this commit it used to be like this:
      
      MULTI
      EXEC
      ... actual commands of the transaction ...
      
      Because after all that is the natural order of things. Transaction
      commands are queued and executed *only after* EXEC is called.
      
      However this makes debugging with MONITOR a mess, so the code was
      modified to provide a coherent output.
      
      What happens is that MULTI is rendered in the MONITOR output as far as
      possible, instead EXEC is propagated only after the transaction is
      executed, or even in the case it fails because of WATCH, so in this case
      you'll simply see:
      
      MULTI
      EXEC
      
      An empty transaction.
      a1b1c1ea
  20. 17 Sep, 2012 1 commit
    • antirez's avatar
      A reimplementation of blocking operation internals. · 7eb850ef
      antirez authored
      Redis provides support for blocking operations such as BLPOP or BRPOP.
      This operations are identical to normal LPOP and RPOP operations as long
      as there are elements in the target list, but if the list is empty they
      block waiting for new data to arrive to the list.
      
      All the clients blocked waiting for th same list are served in a FIFO
      way, so the first that blocked is the first to be served when there is
      more data pushed by another client into the list.
      
      The previous implementation of blocking operations was conceived to
      serve clients in the context of push operations. For for instance:
      
      1) There is a client "A" blocked on list "foo".
      2) The client "B" performs `LPUSH foo somevalue`.
      3) The client "A" is served in the context of the "B" LPUSH,
      synchronously.
      
      Processing things in a synchronous way was useful as if "A" pushes a
      value that is served by "B", from the point of view of the database is a
      NOP (no operation) thing, that is, nothing is replicated, nothing is
      written in the AOF file, and so forth.
      
      However later we implemented two things:
      
      1) Variadic LPUSH that could add multiple values to a list in the
      context of a single call.
      2) BRPOPLPUSH that was a version of BRPOP that also provided a "PUSH"
      side effect when receiving data.
      
      This forced us to make the synchronous implementation more complex. If
      client "B" is waiting for data, and "A" pushes three elemnents in a
      single call, we needed to propagate an LPUSH with a missing argument
      in the AOF and replication link. We also needed to make sure to
      replicate the LPUSH side of BRPOPLPUSH, but only if in turn did not
      happened to serve another blocking client into another list ;)
      
      This were complex but with a few of mutually recursive functions
      everything worked as expected... until one day we introduced scripting
      in Redis.
      
      Scripting + synchronous blocking operations = Issue #614.
      
      Basically you can't "rewrite" a script to have just a partial effect on
      the replicas and AOF file if the script happened to serve a few blocked
      clients.
      
      The solution to all this problems, implemented by this commit, is to
      change the way we serve blocked clients. Instead of serving the blocked
      clients synchronously, in the context of the command performing the PUSH
      operation, it is now an asynchronous and iterative process:
      
      1) If a key that has clients blocked waiting for data is the subject of
      a list push operation, We simply mark keys as "ready" and put it into a
      queue.
      2) Every command pushing stuff on lists, as a variadic LPUSH, a script,
      or whatever it is, is replicated verbatim without any rewriting.
      3) Every time a Redis command, a MULTI/EXEC block, or a script,
      completed its execution, we run the list of keys ready to serve blocked
      clients (as more data arrived), and process this list serving the
      blocked clients.
      4) As a result of "3" maybe more keys are ready again for other clients
      (as a result of BRPOPLPUSH we may have push operations), so we iterate
      back to step "3" if it's needed.
      
      The new code has a much simpler semantics, and a simpler to understand
      implementation, with the disadvantage of not being able to "optmize out"
      a PUSH+BPOP as a No OP.
      
      This commit will be tested with care before the final merge, more tests
      will be added likely.
      7eb850ef
  21. 04 Sep, 2012 1 commit
    • antirez's avatar
      Scripting: Force SORT BY constant determinism inside SORT itself. · 36741b2c
      antirez authored
      SORT is able to return (faster than when ordering) unordered output if
      the "BY" clause is used with a constant value. However we try to play
      well with scripting requirements of determinism providing always sorted
      outputs when SORT (and other similar commands) are called by Lua
      scripts.
      
      However we used the general mechanism in place in scripting in order to
      reorder SORT output, that is, if the command has the "S" flag set, the
      Lua scripting engine will take an additional step when converting a
      multi bulk reply to Lua value, calling a Lua sorting function.
      
      This is suboptimal as we can do it faster inside SORT itself.
      This is also broken as issue #545 shows us: basically when SORT is used
      with a constant BY, and additionally also GET is used, the Lua scripting
      engine was trying to order the output as a flat array, while it was
      actually a list of key-value pairs.
      
      What we do know is to recognized if the caller of SORT is the Lua client
      (since we can check this using the REDIS_LUA_CLIENT flag). If so, and if
      a "don't sort" condition is triggered by the BY option with a constant
      string, we force the lexicographical sorting.
      
      This commit fixes this bug and improves the performance, and at the same
      time simplifies the implementation. This does not mean I'm smart today,
      it means I was stupid when I committed the original implementation ;)
      36741b2c
  22. 02 Sep, 2012 1 commit
    • antirez's avatar
      Send an async PING before starting replication with master. · bb66fc31
      antirez authored
      During the first synchronization step of the replication process, a Redis
      slave connects with the master in a non blocking way. However once the
      connection is established the replication continues sending the REPLCONF
      command, and sometimes the AUTH command if needed. Those commands are
      send in a partially blocking way (blocking with timeout in the order of
      seconds).
      
      Because it is common for a blocked master to accept connections even if
      it is actually not able to reply to the slave requests, it was easy for
      a slave to block if the master had serious issues, but was still able to
      accept connections in the listening socket.
      
      For this reason we now send an asynchronous PING request just after the
      non blocking connection ended in a successful way, and wait for the
      reply before to continue with the replication process. It is very
      unlikely that a master replying to PING can't reply to the other
      commands.
      
      This solution was proposed by Didier Spezia (Thanks!) so that we don't
      need to turn all the replication process into a non blocking affair, but
      still the probability of a slave blocked is minimal even in the event of
      a failing master.
      
      Also we now use getsockopt(SO_ERROR) in order to check errors ASAP
      in the event handler, instead of waiting for actual I/O to return an
      error.
      
      This commit fixes issue #632.
      bb66fc31
  23. 28 Aug, 2012 2 commits
    • antirez's avatar
      Sentinel: Redis-side support for slave priority. · 169a44cb
      antirez authored
      A Redis slave can now be configured with a priority, that is an integer
      number that is shown in INFO output and can be get and set using the
      redis.conf file or the CONFIG GET/SET command.
      
      This field is used by Sentinel during slave election. A slave with lower
      priority is preferred. A slave with priority zero is never elected (and
      is considered to be impossible to elect even if it is the only slave
      available).
      
      A next commit will add support in the Sentinel side as well.
      169a44cb
    • antirez's avatar
      Incrementally flush RDB on disk while loading it from a master. · 784b9308
      antirez authored
      This fixes issue #539.
      
      Basically if there is enough free memory the OS may buffer the RDB file
      that the slave transfers on disk from the master. The file may
      actually be flused on disk at once by the operating system when it gets
      closed by Redis, causing the close system call to block for a long time.
      
      This patch is a modified version of one provided by yoav-steinberg of
      @garantiadata (the original version was posted in the issue #539
      comments), and tries to flush the OS buffers incrementally (every 8 MB
      of loaded data).
      784b9308
  24. 23 Jul, 2012 1 commit
  25. 22 Jul, 2012 1 commit
    • antirez's avatar
      Allow Pub/Sub in contexts where other commands are blocked. · 5d73073f
      antirez authored
      Redis loading data from disk, and a Redis slave disconnected from its
      master with serve-stale-data disabled, are two conditions where
      commands are normally refused by Redis, returning an error.
      
      However there is no reason to disable Pub/Sub commands as well, given
      that this layer does not interact with the dataset. To allow Pub/Sub in
      as many contexts as possible is especially interesting now that Redis
      Sentinel uses Pub/Sub of a Redis master as a communication channel
      between Sentinels.
      
      This commit allows Pub/Sub to be used in the above two contexts where
      it was previously denied.
      5d73073f
  26. 17 Jul, 2012 1 commit
    • Saj Goonatilleke's avatar
      New in INFO: aof_last_bgrewrite_status · 48553a29
      Saj Goonatilleke authored
      Behaves like rdb_last_bgsave_status -- even down to reporting 'ok' when
      no rewrite has been done yet.  (You might want to check that
      aof_last_rewrite_time_sec is not -1.)
      48553a29
  27. 05 Jul, 2012 1 commit
    • Saj Goonatilleke's avatar
      Bug fix: slaves being pinged every second · 9edfe635
      Saj Goonatilleke authored
      REDIS_REPL_PING_SLAVE_PERIOD controls how often the master should
      transmit a heartbeat (PING) to its slaves.  This period, which defaults
      to 10, is measured in seconds.
      
      Redis 2.4 masters used to ping their slaves every ten seconds, just like
      it says on the tin.
      
      The Redis 2.6 masters I have been experimenting with, on the other hand,
      ping their slaves *every second*.  (master_last_io_seconds_ago never
      approaches 10.)  I think the ping period was inadvertently slashed to
      one-tenth of its nominal value around the time REDIS_HZ was introduced.
      This commit reintroduces correct ping schedule behaviour.
      9edfe635
  28. 27 Jun, 2012 1 commit
    • antirez's avatar
      REPLCONF internal command introduced. · 3a328978
      antirez authored
      The REPLCONF command is an internal command (not designed to be directly
      used by normal clients) that allows a slave to set some replication
      related state in the master before issuing SYNC to start the
      replication.
      
      The initial motivation for this command, and the only reason currently
      it is used by the implementation, is to let the slave instance
      communicate its listening port to the slave, so that the master can
      show all the slaves with their listening ports in the "replication"
      section of the INFO output.
      
      This allows clients to auto discover and query all the slaves attached
      into a master.
      
      Currently only a single option of the REPLCONF command is supported, and
      it is called "listening-port", so the slave now starts the replication
      process with something like the following chat:
      
          REPLCONF listening-prot 6380
          SYNC
      
      Note that this works even if the master is an older version of Redis and
      does not understand REPLCONF, because the slave ignores the REPLCONF
      error.
      
      In the future REPLCONF can be used for partial replication and other
      replication related features where there is the need to exchange
      information between master and slave.
      
      NOTE: This commit also fixes a bug: the INFO outout already carried
      information about slaves, but the port was broken, and was obtained
      with getpeername(2), so it was actually just the ephemeral port used
      by the slave to connect to the master as a client.
      3a328978