1. 16 Dec, 2016 1 commit
  2. 14 Dec, 2016 1 commit
    • antirez's avatar
      MIGRATE: Remove upfront ttl initialization. · b53e73e1
      antirez authored
      After the fix for #3673 the ttl var is always initialized inside the
      loop itself, so the early initialization is not needed.
      
      Variables declaration also moved to a more local scope.
      b53e73e1
  3. 13 Dec, 2016 4 commits
    • antirez's avatar
      Writable slaves expires: fix leak in key tracking. · b6f871cf
      antirez authored
      We need to use a dictionary type that frees the key, since we copy the
      keys in the dictionary we use to track expires created in the slave
      side.
      b6f871cf
    • antirez's avatar
      INFO: show num of slave-expires keys tracked. · d1adc85a
      antirez authored
      d1adc85a
    • antirez's avatar
      Fix created->created typo in expire.c · 5b9ba264
      antirez authored
      5b9ba264
    • antirez's avatar
      Replication: fix the infamous key leakage of writable slaves + EXPIRE. · 04542cff
      antirez authored
      BACKGROUND AND USE CASEj
      
      Redis slaves are normally write only, however the supprot a "writable"
      mode which is very handy when scaling reads on slaves, that actually
      need write operations in order to access data. For instance imagine
      having slaves replicating certain Sets keys from the master. When
      accessing the data on the slave, we want to peform intersections between
      such Sets values. However we don't want to intersect each time: to cache
      the intersection for some time often is a good idea.
      
      To do so, it is possible to setup a slave as a writable slave, and
      perform the intersection on the slave side, perhaps setting a TTL on the
      resulting key so that it will expire after some time.
      
      THE BUG
      
      Problem: in order to have a consistent replication, expiring of keys in
      Redis replication is up to the master, that synthesize DEL operations to
      send in the replication stream. However slaves logically expire keys
      by hiding them from read attempts from clients so that if the master did
      not promptly sent a DEL, the client still see logically expired keys
      as non existing.
      
      Because slaves don't actively expire keys by actually evicting them but
      just masking from the POV of read operations, if a key is created in a
      writable slave, and an expire is set, the key will be leaked forever:
      
      1. No DEL will be received from the master, which does not know about
      such a key at all.
      
      2. No eviction will be performed by the slave, since it needs to disable
      eviction because it's up to masters, otherwise consistency of data is
      lost.
      
      THE FIX
      
      In order to fix the problem, the slave should be able to tag keys that
      were created in the slave side and have an expire set in some way.
      
      My solution involved using an unique additional dictionary created by
      the writable slave only if needed. The dictionary is obviously keyed by
      the key name that we need to track: all the keys that are set with an
      expire directly by a client writing to the slave are tracked.
      
      The value in the dictionary is a bitmap of all the DBs where such a key
      name need to be tracked, so that we can use a single dictionary to track
      keys in all the DBs used by the slave (actually this limits the solution
      to the first 64 DBs, but the default with Redis is to use 16 DBs).
      
      This solution allows to pay both a small complexity and CPU penalty,
      which is zero when the feature is not used, actually. The slave-side
      eviction is encapsulated in code which is not coupled with the rest of
      the Redis core, if not for the hook to track the keys.
      
      TODO
      
      I'm doing the first smoke tests to see if the feature works as expected:
      so far so good. Unit tests should be added before merging into the
      4.0 branch.
      04542cff
  4. 12 Dec, 2016 1 commit
  5. 08 Dec, 2016 1 commit
    • Jan-Erik Rediger's avatar
      Reset the ttl for additional keys · 2a32f037
      Jan-Erik Rediger authored
      Before, if a previous key had a TTL set but the current one didn't, the
      TTL was reused and thus resulted in wrong expirations set.
      
      This behaviour was experienced, when `MigrateDefaultPipeline` in
      redis-trib was set to >1
      
      Fixes #3655
      2a32f037
  6. 06 Dec, 2016 1 commit
  7. 05 Dec, 2016 3 commits
    • antirez's avatar
      Modules: types doc updated to new API. · 16cce320
      antirez authored
      16cce320
    • antirez's avatar
      Modules: API doc updated (auto generated). · 37b6e16a
      antirez authored
      37b6e16a
    • antirez's avatar
      Geo: fix computation of bounding box. · 001138ae
      antirez authored
      A bug was reported in the context in issue #3631. The root cause of the
      bug was that certain neighbor boxes were zeroed after the "inside the
      bounding box or not" check, simply because the bounding box computation
      function was wrong.
      
      A few debugging infos where enhanced and moved in other parts of the
      code. A check to avoid steps=0 was added, but is unrelated to this
      issue and I did not verified it was an actual bug in practice.
      001138ae
  8. 02 Dec, 2016 1 commit
  9. 01 Dec, 2016 1 commit
  10. 30 Nov, 2016 3 commits
  11. 29 Nov, 2016 1 commit
  12. 28 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: stop sending newlines to sub-slaves when master is down. · eab865a0
      antirez authored
      This actually includes two changes:
      
      1) No newlines to take the master-slave link up when the upstream master
      is down. Doing this is dangerous because the sub-slave often is received
      replication protocol for an half-command, so can't receive newlines
      without desyncing the replication link, even with the code in order to
      cancel out the bytes that PSYNC2 was using. Moreover this is probably
      also not needed/sane, because anyway the slave can keep serving
      requests, and because if it's configured to don't serve stale data, it's
      a good idea, actually, to break the link.
      
      2) When a +CONTINUE with a different ID is received, we now break
      connection with the sub-slaves: they need to be notified as well. This
      was part of the original specification but for some reason it was not
      implemented in the code, and was alter found as a PSYNC2 bug in the
      integration testing.
      eab865a0
  13. 25 Nov, 2016 1 commit
  14. 24 Nov, 2016 2 commits
  15. 23 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: bugfixing pre release. · 5b7d42ff
      antirez authored
      1. Master replication offset was cleared after switching configuration
      to some other slave, since it was assumed you can't PSYNC after a
      switch. Note the case anymore and when we successfully PSYNC we need to
      have our offset untouched.
      
      2. Secondary replication ID was not reset to "000..." pattern at
      startup.
      
      3. Master in error state replying -LOADING or other transient errors
      forced the slave to discard the cached master and full resync. This is
      now fixed.
      
      4. Better logging of what's happening on failed PSYNCs.
      5b7d42ff
  16. 16 Nov, 2016 2 commits
  17. 10 Nov, 2016 1 commit
    • antirez's avatar
      PSYNC2: Save replication ID/offset on RDB file. · 28c96d73
      antirez authored
      This means that stopping a slave and restarting it will still make it
      able to PSYNC with the master. Moreover the master itself will retain
      its ID/offset, in case it gets turned into a slave, or if a slave will
      try to PSYNC with it with an exactly updated offset (otherwise there is
      no backlog).
      
      This change was possible thanks to PSYNC v2 that makes saving the current
      replication state much simpler.
      28c96d73
  18. 09 Nov, 2016 2 commits
    • antirez's avatar
      PSYNC2: Wrap debugging code with if(0) · 4e5e366e
      antirez authored
      4e5e366e
    • antirez's avatar
      PSYNC2: different improvements to Redis replication. · 2669fb83
      antirez authored
      The gist of the changes is that now, partial resynchronizations between
      slaves and masters (without the need of a full resync with RDB transfer
      and so forth), work in a number of cases when it was impossible
      in the past. For instance:
      
      1. When a slave is promoted to mastrer, the slaves of the old master can
      partially resynchronize with the new master.
      
      2. Chained slalves (slaves of slaves) can be moved to replicate to other
      slaves or the master itsef, without requiring a full resync.
      
      3. The master itself, after being turned into a slave, is able to
      partially resynchronize with the new master, when it joins replication
      again.
      
      In order to obtain this, the following main changes were operated:
      
      * Slaves also take a replication backlog, not just masters.
      
      * Same stream replication for all the slaves and sub slaves. The
      replication stream is identical from the top level master to its slaves
      and is also the same from the slaves to their sub-slaves and so forth.
      This means that if a slave is later promoted to master, it has the
      same replication backlong, and can partially resynchronize with its
      slaves (that were previously slaves of the old master).
      
      * A given replication history is no longer identified by the `runid` of
      a Redis node. There is instead a `replication ID` which changes every
      time the instance has a new history no longer coherent with the past
      one. So, for example, slaves publish the same replication history of
      their master, however when they are turned into masters, they publish
      a new replication ID, but still remember the old ID, so that they are
      able to partially resynchronize with slaves of the old master (up to a
      given offset).
      
      * The replication protocol was slightly modified so that a new extended
      +CONTINUE reply from the master is able to inform the slave of a
      replication ID change.
      
      * REPLCONF CAPA is used in order to notify masters that a slave is able
      to understand the new +CONTINUE reply.
      
      * The RDB file was extended with an auxiliary field that is able to
      select a given DB after loading in the slave, so that the slave can
      continue receiving the replication stream from the point it was
      disconnected without requiring the master to insert "SELECT" statements.
      This is useful in order to guarantee the "same stream" property, because
      the slave must be able to accumulate an identical backlog.
      
      * Slave pings to sub-slaves are now sent in a special form, when the
      top-level master is disconnected, in order to don't interfer with the
      replication stream. We just use out of band "\n" bytes as in other parts
      of the Redis protocol.
      
      An old design document is available here:
      
      https://gist.github.com/antirez/ae068f95c0d084891305
      
      However the implementation is not identical to the description because
      during the work to implement it, different changes were needed in order
      to make things working well.
      2669fb83
  19. 02 Nov, 2016 1 commit
  20. 31 Oct, 2016 2 commits
  21. 28 Oct, 2016 2 commits
  22. 21 Oct, 2016 1 commit
  23. 19 Oct, 2016 1 commit
  24. 17 Oct, 2016 2 commits
    • antirez's avatar
    • Pedro Melo's avatar
      Fixes compilation on MacOS 10.8.5, Clang tags/Apple/clang-421.0.57 · 2000abc8
      Pedro Melo authored
      Redis fails to compile on MacOS 10.8.5 with Clang 4, version 421.0.57
      (based on LLVM 3.1svn).
      
      When compiling zmalloc.c, we get these warnings:
      
              CC zmalloc.o
          zmalloc.c:109:5: warning: implicit declaration of function '__atomic_add_fetch' is invalid in C99 [-Wimplicit-function-declaration]
              update_zmalloc_stat_alloc(zmalloc_size(ptr));
              ^
          zmalloc.c:75:9: note: expanded from macro 'update_zmalloc_stat_alloc'
                  atomicIncr(used_memory,__n,used_memory_mutex); \
                  ^
          ./atomicvar.h:57:37: note: expanded from macro 'atomicIncr'
          #define atomicIncr(var,count,mutex) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
                                              ^
          zmalloc.c:145:5: warning: implicit declaration of function '__atomic_sub_fetch' is invalid in C99 [-Wimplicit-function-declaration]
              update_zmalloc_stat_free(oldsize);
              ^
          zmalloc.c:85:9: note: expanded from macro 'update_zmalloc_stat_free'
                  atomicDecr(used_memory,__n,used_memory_mutex); \
                  ^
          ./atomicvar.h:58:37: note: expanded from macro 'atomicDecr'
          #define atomicDecr(var,count,mutex) __atomic_sub_fetch(&var,(count),__ATOMIC_RELAXED)
                                              ^
          zmalloc.c:205:9: warning: implicit declaration of function '__atomic_load_n' is invalid in C99 [-Wimplicit-function-declaration]
                  atomicGet(used_memory,um,used_memory_mutex);
                  ^
          ./atomicvar.h:60:14: note: expanded from macro 'atomicGet'
              dstvar = __atomic_load_n(&var,__ATOMIC_RELAXED); \
                       ^
          3 warnings generated.
      
      Also on lazyfree.c:
      
              CC lazyfree.o
          lazyfree.c:68:13: warning: implicit declaration of function '__atomic_add_fetch' is invalid in C99 [-Wimplicit-function-declaration]
                      atomicIncr(lazyfree_objects,1,lazyfree_objects_mutex);
                      ^
          ./atomicvar.h:57:37: note: expanded from macro 'atomicIncr'
          #define atomicIncr(var,count,mutex) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
                                              ^
          lazyfree.c:111:5: warning: implicit declaration of function '__atomic_sub_fetch' is invalid in C99 [-Wimplicit-function-declaration]
              atomicDecr(lazyfree_objects,1,lazyfree_objects_mutex);
              ^
          ./atomicvar.h:58:37: note: expanded from macro 'atomicDecr'
          #define atomicDecr(var,count,mutex) __atomic_sub_fetch(&var,(count),__ATOMIC_RELAXED)
                                              ^
          2 warnings generated.
      
      Then in the linking stage:
      
              LINK redis-server
          Undefined symbols for architecture x86_64:
            "___atomic_add_fetch", referenced from:
                _zmalloc in zmalloc.o
                _zcalloc in zmalloc.o
                _zrealloc in zmalloc.o
                _dbAsyncDelete in lazyfree.o
                _emptyDbAsync in lazyfree.o
                _slotToKeyFlushAsync in lazyfree.o
            "___atomic_load_n", referenced from:
                _zmalloc_used_memory in zmalloc.o
                _zmalloc_get_fragmentation_ratio in zmalloc.o
            "___atomic_sub_fetch", referenced from:
                _zrealloc in zmalloc.o
                _zfree in zmalloc.o
                _lazyfreeFreeObjectFromBioThread in lazyfree.o
                _lazyfreeFreeDatabaseFromBioThread in lazyfree.o
                _lazyfreeFreeSlotsMapFromBioThread in lazyfree.o
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          make[1]: *** [redis-server] Error 1
          make: *** [all] Error 2
      
      With this patch, the compilation is sucessful, no warnings.
      
      Running `make test` we get a almost clean bill of health. Test pass with
      one exception:
      
          [err]: Check for memory leaks (pid 52793) in tests/unit/dump.tcl
          [err]: Check for memory leaks (pid 53103) in tests/unit/auth.tcl
          [err]: Check for memory leaks (pid 53117) in tests/unit/auth.tcl
          [err]: Check for memory leaks (pid 53131) in tests/unit/protocol.tcl
          [err]: Check for memory leaks (pid 53145) in tests/unit/protocol.tcl
          [ok]: Check for memory leaks (pid 53160)
          [err]: Check for memory leaks (pid 53175) in tests/unit/scan.tcl
          [ok]: Check for memory leaks (pid 53189)
          [err]: Check for memory leaks (pid 53221) in tests/unit/type/incr.tcl
          .
          .
          .
      
      Full debug log (289MB, uncompressed) available at
      https://dl.dropboxusercontent.com/u/75548/logs/redis-debug-log-macos-10.8.5.log.xz
      
      
      
      Most if not all of the memory leak tests fail. Not sure if this is
      related. They are the only ones that fail. I belive they are not related,
      but just the memory leak detector is not working properly on 10.8.5.
      Signed-off-by: default avatarPedro Melo <melo@simplicidade.org>
      2000abc8
  25. 14 Oct, 2016 1 commit
    • antirez's avatar
      SWAPDB command. · c7a4e694
      antirez authored
      This new command swaps two Redis databases, so that immediately all the
      clients connected to a given DB will see the data of the other DB, and
      the other way around. Example:
      
          SWAPDB 0 1
      
      This will swap DB 0 with DB 1. All the clients connected with DB 0 will
      immediately see the new data, exactly like all the clients connected
      with DB 1 will see the data that was formerly of DB 0.
      
      MOTIVATION AND HISTORY
      ---
      
      The command was recently demanded by Pedro Melo, but was suggested in
      the past multiple times, and always refused by me.
      
      The reason why it was asked: Imagine you have clients operating in DB 0.
      At the same time, you create a new version of the dataset in DB 1.
      When the new version of the dataset is available, you immediately want
      to swap the two views, so that the clients will transparently use the
      new version of the data. At the same time you'll likely destroy the
      DB 1 dataset (that contains the old data) and start to build a new
      version, to repeat the process.
      
      This is an interesting pattern, but the reason why I always opposed to
      implement this, was that FLUSHDB was a blocking command in Redis before
      Redis 4.0 improvements. Now we have FLUSHDB ASYNC that releases the
      old data in O(1) from the point of view of the client, to reclaim memory
      incrementally in a different thread.
      
      At this point, the pattern can really be supported without latency
      spikes, so I'm providing this implementation for the users to comment.
      In case a very compelling argument will be made against this new command
      it may be removed.
      
      BEHAVIOR WITH BLOCKING OPERATIONS
      ---
      
      If a client is blocking for a list in a given DB, after the swap it will
      still be blocked in the same DB ID, since this is the most logical thing
      to do: if I was blocked for a list push to list "foo", even after the
      swap I want still a LPUSH to reach the key "foo" in the same DB in order
      to unblock.
      
      However an interesting thing happens when a client is, for instance,
      blocked waiting for new elements in list "foo" of DB 0. Then the DB
      0 and 1 are swapped with SWAPDB. However the DB 1 happened to have
      a list called "foo" containing elements. When this happens, this
      implementation can correctly unblock the client.
      
      It is possible that there are subtle corner cases that are not covered
      in the implementation, but since the command is self-contained from the
      POV of the implementation and the Redis core, it cannot cause anything
      bad if not used.
      
      Tests and documentation are yet to be provided.
      c7a4e694
  26. 13 Oct, 2016 2 commits