1. 30 Mar, 2018 1 commit
  2. 29 Mar, 2018 1 commit
  3. 25 Mar, 2018 1 commit
    • antirez's avatar
      AOF: enable RDB-preamble rewriting by default. · 28d28ef3
      antirez authored
      There are too many advantages in doing this, RDB is faster to persist,
      more compact, much faster to load back. The main issues here are that
      the code is less tested because this was not the old default (so we are
      enabling it for the new 5.0 release), and that the AOF is no longer a
      trivially parsable format from now on. However the non-preamble mode
      will be supported in the future as well, if new data types will be
      added.
      28d28ef3
  4. 19 Mar, 2018 1 commit
  5. 15 Mar, 2018 6 commits
  6. 14 Mar, 2018 1 commit
    • antirez's avatar
      Cluster: ability to prevent slaves from failing over their masters. · 432bf477
      antirez authored
      This commit, in some parts derived from PR #3041 which is no longer
      possible to merge (because the user deleted the original branch),
      implements the ability of slaves to have a special configuration
      preventing that they try to start a failover when the master is failing.
      
      There are multiple reasons for wanting this, and the feautre was
      requested in issue #3021 time ago.
      
      The differences between this patch and the original PR are the
      following:
      
      1. The flag is saved/loaded on the nodes configuration.
      2. The 'myself' node is now flag-aware, the flag is updated as needed
         when the configuration is changed via CONFIG SET.
      3. The flag name uses NOFAILOVER instead of NO_FAILOVER to be consistent
         with existing NOADDR.
      4. The redis.conf documentation was rewritten.
      
      Thanks to @deep011 for the original patch.
      432bf477
  7. 12 Mar, 2018 2 commits
    • Oran Agra's avatar
      Adding real allocator fragmentation to INFO and MEMORY command + active defrag test · 806736cd
      Oran Agra authored
      other fixes / improvements:
      - LUA script memory isn't taken from zmalloc (taken from libc malloc)
        so it can cause high fragmentation ratio to be displayed (which is false)
      - there was a problem with "fragmentation" info being calculated from
        RSS and used_memory sampled at different times (now sampling them together)
      
      other details:
      - adding a few more allocator info fields to INFO and MEMORY commands
      - improve defrag test to measure defrag latency of big keys
      - increasing the accuracy of the defrag test (by looking at real grag info)
        this way we can use an even lower threshold and still avoid false positives
      - keep the old (total) "fragmentation" field unchanged, but add new ones for spcific things
      - add these the MEMORY DOCTOR command
      - deduct LUA memory from the rss in case of non jemalloc allocator (one for which we don't "allocator active/used")
      - reduce sampling rate of the rss and allocator info
      806736cd
    • Oran Agra's avatar
      active defrag v2 · be1b4aa9
      Oran Agra authored
      - big keys are not defragged in one go from within the dict scan
        instead they are scanned in parts after the main dict hash bucket is done.
      - add latency monitor sample for defrag
      - change default active-defrag-cycle-min to induce lower latency
      - make active defrag start a new scan right away if needed, so it's easier
        (for the test suite) to detect when it's done
      - make active defrag quick the current cycle after each db / big key
      - defrag  some non key long term global allocations
      - some refactoring for smaller functions and more reusable code
      - during dict rehashing, one scan iteration of the dict, can end up scanning
        one bucket in the smaller dict and many many buckets in the larger dict.
        so waiting for 16 scan iterations before checking the time, may be much too long.
      be1b4aa9
  8. 19 Feb, 2018 1 commit
    • antirez's avatar
      Track number of logically expired keys still in memory. · ffde73c5
      antirez authored
      This commit adds two new fields in the INFO output, stats section:
      
      expired_stale_perc:0.34
      expired_time_cap_reached_count:58
      
      The first field is an estimate of the number of keys that are yet in
      memory but are already logically expired. They reason why those keys are
      yet not reclaimed is because the active expire cycle can't spend more
      time on the process of reclaiming the keys, and at the same time nobody
      is accessing such keys. However as the active expire cycle runs, while
      it will eventually have to return to the caller, because of time limit
      or because there are less than 25% of keys logically expired in each
      given database, it collects the stats in order to populate this INFO
      field.
      
      Note that expired_stale_perc is a running average, where the current
      sample accounts for 5% and the history for 95%, so you'll see it
      changing smoothly over time.
      
      The other field, expired_time_cap_reached_count, counts the number
      of times the expire cycle had to stop, even if still it was finding a
      sizeable number of keys yet to expire, because of the time limit.
      This allows people handling operations to understand if the Redis
      server, during mass-expiration events, is able to collect keys fast
      enough usually. It is normal for this field to increment during mass
      expires, but normally it should very rarely increment. When instead it
      constantly increments, it means that the current workloads is using
      a very important percentage of CPU time to expire keys.
      
      This feature was created thanks to the hints of Rashmi Ramesh and
      Bart Robinson from Twitter. In private email exchanges, they noted how
      it was important to improve the observability of this parameter in the
      Redis server. Actually in big deployments, the amount of keys that are
      yet to expire in each server, even if they are logically expired, may
      account for a very big amount of wasted memory.
      ffde73c5
  9. 14 Feb, 2018 2 commits
  10. 11 Jan, 2018 1 commit
  11. 29 Dec, 2017 1 commit
  12. 05 Dec, 2017 1 commit
    • antirez's avatar
      add linkClient(): adds the client and caches the list node. · 62a4b817
      antirez authored
      We have this operation in two places: when caching the master and
      when linking a new client after the client creation. By having an API
      for this we avoid incurring in errors when modifying one of the two
      places forgetting the other. The function is also a good place where to
      document why we cache the linked list node.
      
      Related to #4497 and #4210.
      62a4b817
  13. 04 Dec, 2017 1 commit
    • antirez's avatar
      Refactoring: improve luaCreateFunction() API. · 60d26acf
      antirez authored
      The function in its initial form, and after the fixes for the PSYNC2
      bugs, required code duplication in multiple spots. This commit modifies
      it in order to always compute the script name independently, and to
      return the SDS of the SHA of the body: this way it can be used in all
      the places, including for SCRIPT LOAD, without duplicating the code to
      create the Lua function name. Note that this requires to re-compute the
      body SHA1 in the case of EVAL seeing a script for the first time, but
      this should not change scripting performance in any way because new
      scripts definition is a rare event happening the first time a script is
      seen, and the SHA1 computation is anyway not a very slow process against
      the typical Redis script and compared to the actua Lua byte compiling of
      the body.
      
      Note that the function used to assert() if a duplicated script was
      loaded, however actually now two times over three, we want the function
      to handle duplicated scripts just fine: this happens in SCRIPT LOAD and
      in RDB AUX "lua" loading. Moreover the assert was not defending against
      some obvious failure mode, so now the function always tests against
      already defined functions at start.
      60d26acf
  14. 01 Dec, 2017 14 commits
  15. 30 Nov, 2017 2 commits
  16. 28 Nov, 2017 1 commit
    • Itamar Haber's avatar
      Standardizes the 'help' subcommand · 59d52f7f
      Itamar Haber authored
      This adds a new `addReplyHelp` helper that's used by commands
      when returning a help text. The following commands have been
      touched: DEBUG, OBJECT, COMMAND, PUBSUB, SCRIPT and SLOWLOG.
      
      WIP
      
      Fix entry command table entry for OBJECT for HELP option.
      
      After #4472 the command may have just 2 arguments.
      
      Improve OBJECT HELP descriptions.
      
      See #4472.
      
      WIP 2
      
      WIP 3
      59d52f7f
  17. 27 Nov, 2017 2 commits
    • zhaozhao.zz's avatar
      LFU: do some changes about LFU to find hotkeys · 583c3147
      zhaozhao.zz authored
      Firstly, use access time to replace the decreas time of LFU.
      For function LFUDecrAndReturn,
      it should only try to get decremented counter,
      not update LFU fields, we will update it in an explicit way.
      And we will times halve the counter according to the times of
      elapsed time than server.lfu_decay_time.
      Everytime a key is accessed, we should update the LFU
      including update access time, and increment the counter after
      call function LFUDecrAndReturn.
      If a key is overwritten, the LFU should be also updated.
      Then we can use `OBJECT freq` command to get a key's frequence,
      and LFUDecrAndReturn should be called in `OBJECT freq` command
      in case of the key has not been accessed for a long time,
      because we update the access time only when the key is read or
      overwritten.
      583c3147
    • zhaozhao.zz's avatar
      LFU: change lfu* parameters to int · 53cea972
      zhaozhao.zz authored
      53cea972
  18. 12 Jul, 2017 1 commit
    • antirez's avatar
      Fix replication of SLAVEOF inside transaction. · e74f0aa6
      antirez authored
      In Redis 4.0 replication, with the introduction of PSYNC2, masters and
      slaves replicate commands to cascading slaves and to the replication
      backlog itself in a different way compared to the past.
      
      Masters actually replicate the effects of client commands.
      Slaves just propagate what they receive from masters.
      
      This mechanism can cause problems when the configuration of an instance
      is changed from master to slave inside a transaction. For instance
      we could send to a master instance the following sequence:
      
          MULTI
          SLAVEOF 127.0.0.1 0
          EXEC
          SLAVEOF NO ONE
      
      Before the fixes in this commit, the MULTI command used to be propagated
      into the replication backlog, however after the SLAVEOF command the
      instance is a slave, so the EXEC implementation failed to also propagate
      the EXEC command. When the slaves of the above instance reconnected,
      they were incrementally synchronized just sending a "MULTI". This put
      the master client (in the slaves) into MULTI state, breaking the
      replication.
      
      Notably even Redis Sentinel uses the above approach in order to guarantee
      that configuration changes are always performed together with rewrites
      of the configuration and with clients disconnection. Sentiel does:
      
          MULTI
          SLAVEOF ...
          CONFIG REWRITE
          CLIENT KILL TYPE normal
          EXEC
      
      So this was a really problematic issue. However even with the fix in
      this commit, that will add the final EXEC to the replication stream in
      case the instance was switched from master to slave during the
      transaction, the result would be to increment the slave replication
      offset, so a successive reconnection with the new master, will not
      permit a successful partial resynchronization: no way the new master can
      provide us with the backlog needed, we incremented our offset to a value
      that the new master cannot have.
      
      However the EXEC implementation waits to emit the MULTI, so that if the
      commands inside the transaction actually do not need to be replicated,
      no commands propagation happens at all. From multi.c:
      
          if (!must_propagate && !(c->cmd->flags & (CMD_READONLY|CMD_ADMIN))) {
      	execCommandPropagateMulti(c);
      	must_propagate = 1;
          }
      
      The above code is already modified by this commit you are reading.
      Now also ADMIN commands do not trigger the emission of MULTI. It is actually
      not clear why we do not just check for CMD_WRITE... Probably I wrote it this
      way in order to make the code more reliable: better to over-emit MULTI
      than not emitting it in time.
      
      So this commit should indeed fix issue #3836 (verified), however it looks
      like some reconsideration of this code path is needed in the long term.
      
      BONUS POINT: The reverse bug.
      
      Even in a read only slave "B", in a replication setup like:
      
      	A -> B -> C
      
      There are commands without the READONLY nor the ADMIN flag, that are also
      not flagged as WRITE commands. An example is just the PING command.
      
      So if we send B the following sequence:
      
          MULTI
          PING
          SLAVEOF NO ONE
          EXEC
      
      The result will be the reverse bug, where only EXEC is emitted, but not the
      previous MULTI. However this apparently does not create problems in practice
      but it is yet another acknowledge of the fact some work is needed here
      in order to make this code path less surprising.
      
      Note that there are many different approaches we could follow. For instance
      MULTI/EXEC blocks containing administrative commands may be allowed ONLY
      if all the commands are administrative ones, otherwise they could be
      denined. When allowed, the commands could simply never be replicated at all.
      e74f0aa6