1. 21 Sep, 2017 2 commits
  2. 20 Sep, 2017 7 commits
    • antirez's avatar
      PSYNC2: More refinements related to #4316. · 79567b6e
      antirez authored
      79567b6e
    • zhaozhao.zz's avatar
      PSYNC2: make persisiting replication info more solid · f1194649
      zhaozhao.zz authored
      This commit is a reinforcement of commit c1c99e9f.
      
      1. Replication information can be stored when the RDB file is
      generated by a mater using server.slaveseldb when server.repl_backlog
      is not NULL, or set repl_stream_db be -1. That's safe, because
      NULL server.repl_backlog will trigger full synchronization,
      then master will send SELECT command to replicaiton stream.
      2. Only do rdbSave* when rsiptr is not NULL,
      if we do rdbSave* without rdbSaveInfo, slave will miss repl-stream-db.
      3. Save the replication informations also in the case of
      SAVE command, FLUSHALL command and DEBUG reload.
      f1194649
    • antirez's avatar
      PSYNC2: Fix the way replication info is saved/loaded from RDB. · 097a5556
      antirez authored
      This commit attempts to fix a number of bugs reported in #4316.
      They are related to the way replication info like replication ID,
      offsets, and currently selected DB in the master client, are stored
      and loaded by Redis. In order to avoid inconsistencies the changes in
      this commit try to enforce that:
      
      1. Replication information are only stored when the RDB file is
      generated by a slave that has a valid 'master' client, so that we can
      always extract the currently selected DB.
      2. When replication informations are persisted in the RDB file, all the
      info for a successful PSYNC or nothing is persisted.
      3. The RDB replication informations are only loaded if the instance is
      configured as a slave, otherwise a master can start with IDs that relate
      to a different history of the data set, and stil retain such IDs in the
      future while receiving unrelated writes.
      097a5556
    • antirez's avatar
      PSYNC2: Create backlog on slave partial sync as well. · f1a2cbfd
      antirez authored
      A slave may be started with an RDB file able to provide enough slave to
      perform a successful partial SYNC with its master. However in such a
      case, how outlined in issue #4268, the slave backlog will not be
      started, since it was only initialized on full syncs attempts. This
      creates different problems with successive PSYNC attempts that will
      always result in full synchronizations.
      
      Thanks to @fdingiit for discovering the issue.
      f1a2cbfd
    • antirez's avatar
      Add MEMORY DOCTOR to MEMORY HELP. · 0c0b77d1
      antirez authored
      0c0b77d1
    • Mota's avatar
      redis-benchmark: default value size usage update. · fa6bd1b2
      Mota authored
      default size of SET/GET value in usage should be 3 bytes as in main code.
      fa6bd1b2
    • jybaek's avatar
      Remove Duplicate Processing · ad0ddcf3
      jybaek authored
      ad0ddcf3
  3. 18 Sep, 2017 1 commit
    • Oran Agra's avatar
      Flush append only buffers before existing. · 8651e5d5
      Oran Agra authored
      when SHUTDOWN command is recived it is possible that some of the recent
      command were not yet flushed from the AOF buffer, and the server
      experiences data loss at shutdown.
      8651e5d5
  4. 02 Aug, 2017 2 commits
  5. 24 Jul, 2017 9 commits
  6. 23 Jul, 2017 4 commits
    • antirez's avatar
      Modules: don't crash when Lua calls a module blocking command. · 41e3617d
      antirez authored
      Lua scripting does not support calling blocking commands, however all
      the native Redis commands are flagged as "s" (no scripting flag), so
      this is not possible at all. With modules there is no such mechanism in
      order to flag a command as non callable by the Lua scripting engine,
      moreover we cannot trust the modules users from complying all the times:
      it is likely that modules will be released to have blocking commands
      without such commands being flagged correctly, even if we provide a way to
      signal this fact.
      
      This commit attempts to address the problem in a short term way, by
      detecting that a module is trying to block in the context of the Lua
      scripting engine client, and preventing to do this. The module will
      actually believe to block as usually, but what happens is that the Lua
      script receives an error immediately, and the background call is ignored
      by the Redis engine (if not for the cleanup callbacks, once it
      unblocks).
      
      Long term, the more likely solution, is to introduce a new call called
      RedisModule_GetClientFlags(), so that a command can detect if the caller
      is a Lua script, and return an error, or avoid blocking at all.
      
      Being the blocking API experimental right now, more work is needed in
      this regard in order to reach a level well blocking module commands and
      all the other Redis subsystems interact peacefully.
      
      Now the effect is like the following:
      
          127.0.0.1:6379> eval "redis.call('hello.block',1,5000)" 0
          (error) ERR Error running script (call to
          f_b5ba35ff97bc1ef23debc4d6e9fd802da187ed53): @user_script:1: ERR
          Blocking module command called from Lua script
      
      This commit fixes issue #4127 in the short term.
      41e3617d
    • antirez's avatar
      10370b20
    • antirez's avatar
      Make representClusterNodeFlags() more robust. · b6c55a89
      antirez authored
      This function failed when an internal-only flag was set as an only flag
      in a node: the string was trimmed expecting a final comma before
      exiting the function, causing a crash. See issue #4142.
      Moreover generation of flags representation only needed at DEBUG log
      level was always performed: a waste of CPU time. This is fixed as well
      by this commit.
      b6c55a89
    • antirez's avatar
      Fix two bugs in moduleTypeLookupModuleByID(). · 9a4f3d72
      antirez authored
      The function cache was not working at all, and the function returned
      wrong values if there where two or more modules exporting native data
      types.
      
      See issue #4131 for more details.
      9a4f3d72
  7. 14 Jul, 2017 13 commits
    • antirez's avatar
      Allow certain modules APIs only defining REDISMODULE_EXPERIMENTAL_API. · 7302e186
      antirez authored
      Those calls may be subject to changes in the future, so the user should
      acknowledge it is using non stable API.
      7302e186
    • antirez's avatar
      Redis 4.0.0 GA. · 05b81d2b
      antirez authored
      05b81d2b
    • antirez's avatar
      Modules: fix thread safe context DB selection. · c29852ff
      antirez authored
      Before this fix the DB currenty selected by the client blocked was not
      respected and operations were always performed on DB 0.
      c29852ff
    • antirez's avatar
      Modules documentation removed from source. · b73f186a
      antirez authored
      Moving to redis-doc repository to publish via Redis.io.
      b73f186a
    • antirez's avatar
      09d93ec9
    • antirez's avatar
      Fix replication of SLAVEOF inside transaction. · 87aabb1a
      antirez authored
      In Redis 4.0 replication, with the introduction of PSYNC2, masters and
      slaves replicate commands to cascading slaves and to the replication
      backlog itself in a different way compared to the past.
      
      Masters actually replicate the effects of client commands.
      Slaves just propagate what they receive from masters.
      
      This mechanism can cause problems when the configuration of an instance
      is changed from master to slave inside a transaction. For instance
      we could send to a master instance the following sequence:
      
          MULTI
          SLAVEOF 127.0.0.1 0
          EXEC
          SLAVEOF NO ONE
      
      Before the fixes in this commit, the MULTI command used to be propagated
      into the replication backlog, however after the SLAVEOF command the
      instance is a slave, so the EXEC implementation failed to also propagate
      the EXEC command. When the slaves of the above instance reconnected,
      they were incrementally synchronized just sending a "MULTI". This put
      the master client (in the slaves) into MULTI state, breaking the
      replication.
      
      Notably even Redis Sentinel uses the above approach in order to guarantee
      that configuration changes are always performed together with rewrites
      of the configuration and with clients disconnection. Sentiel does:
      
          MULTI
          SLAVEOF ...
          CONFIG REWRITE
          CLIENT KILL TYPE normal
          EXEC
      
      So this was a really problematic issue. However even with the fix in
      this commit, that will add the final EXEC to the replication stream in
      case the instance was switched from master to slave during the
      transaction, the result would be to increment the slave replication
      offset, so a successive reconnection with the new master, will not
      permit a successful partial resynchronization: no way the new master can
      provide us with the backlog needed, we incremented our offset to a value
      that the new master cannot have.
      
      However the EXEC implementation waits to emit the MULTI, so that if the
      commands inside the transaction actually do not need to be replicated,
      no commands propagation happens at all. From multi.c:
      
          if (!must_propagate && !(c->cmd->flags & (CMD_READONLY|CMD_ADMIN))) {
      	execCommandPropagateMulti(c);
      	must_propagate = 1;
          }
      
      The above code is already modified by this commit you are reading.
      Now also ADMIN commands do not trigger the emission of MULTI. It is actually
      not clear why we do not just check for CMD_WRITE... Probably I wrote it this
      way in order to make the code more reliable: better to over-emit MULTI
      than not emitting it in time.
      
      So this commit should indeed fix issue #3836 (verified), however it looks
      like some reconsideration of this code path is needed in the long term.
      
      BONUS POINT: The reverse bug.
      
      Even in a read only slave "B", in a replication setup like:
      
      	A -> B -> C
      
      There are commands without the READONLY nor the ADMIN flag, that are also
      not flagged as WRITE commands. An example is just the PING command.
      
      So if we send B the following sequence:
      
          MULTI
          PING
          SLAVEOF NO ONE
          EXEC
      
      The result will be the reverse bug, where only EXEC is emitted, but not the
      previous MULTI. However this apparently does not create problems in practice
      but it is yet another acknowledge of the fact some work is needed here
      in order to make this code path less surprising.
      
      Note that there are many different approaches we could follow. For instance
      MULTI/EXEC blocks containing administrative commands may be allowed ONLY
      if all the commands are administrative ones, otherwise they could be
      denined. When allowed, the commands could simply never be replicated at all.
      87aabb1a
    • antirez's avatar
      CLUSTER GETKEYSINSLOT: avoid overallocating. · 44f89d1d
      antirez authored
      Close #3911.
      44f89d1d
    • antirez's avatar
      Fix isHLLObjectOrReply() to handle integer encoded strings. · 0df24b68
      antirez authored
      Close #3766.
      0df24b68
    • antirez's avatar
      Clients blocked in modules: free argv/argc later. · 884ceb69
      antirez authored
      See issue #3844 for more information.
      884ceb69
    • antirez's avatar
      Event loop: call after sleep() only from top level. · ccbdd762
      antirez authored
      In general we do not want before/after sleep() callbacks to be called
      when we re-enter the event loop, since those calls are only designed in
      order to perform operations every main iteration of the event loop, and
      re-entering is often just a way to incrementally serve clietns with
      error messages or other auxiliary operations. However, if we call the
      callbacks, we are then forced to think at before/after sleep callbacks
      as re-entrant, which is much harder without any good need.
      
      However here there was also a clear bug: beforeSleep() was actually
      never called when re-entering the event loop. But the new afterSleep()
      callback was. This is broken and in this instance re-entering
      afterSleep() caused a modules GIL dead lock.
      ccbdd762
    • antirez's avatar
      10925e46
    • Guy Benoish's avatar
      99bb1c74
    • antirez's avatar
      cfdcd440
  8. 06 Jul, 2017 2 commits