1. 31 Jul, 2020 1 commit
  2. 20 Jul, 2020 1 commit
  3. 14 Jul, 2020 1 commit
    • Oran Agra's avatar
      diskless master disconnect replicas when rdb child failed (#7518) · a176cb56
      Oran Agra authored
      in case the rdb child failed, crashed or terminated unexpectedly redis
      would have marked the replica clients with repl_put_online_on_ack and
      then kill them only after a minute when no ack was received.
      
      it would not stream anything to these connections, so the only effect of
      this bug is a delay of 1 minute in the replicas attempt to re-connect.
      a176cb56
  4. 06 Jun, 2020 2 commits
  5. 28 May, 2020 1 commit
  6. 27 May, 2020 2 commits
  7. 26 May, 2020 2 commits
  8. 25 May, 2020 2 commits
  9. 22 May, 2020 3 commits
    • antirez's avatar
      Make disconnectSlaves() synchronous in the base case. · adc5df1b
      antirez authored
      Otherwise we run into that:
      
      Backtrace:
      src/redis-server 127.0.0.1:21322(logStackTrace+0x45)[0x479035]
      src/redis-server 127.0.0.1:21322(sigsegvHandler+0xb9)[0x4797f9]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fd373c5e390]
      src/redis-server 127.0.0.1:21322(_serverAssert+0x6a)[0x47660a]
      src/redis-server 127.0.0.1:21322(freeReplicationBacklog+0x42)[0x451282]
      src/redis-server 127.0.0.1:21322[0x4552d4]
      src/redis-server 127.0.0.1:21322[0x4c5593]
      src/redis-server 127.0.0.1:21322(aeProcessEvents+0x2e6)[0x42e786]
      src/redis-server 127.0.0.1:21322(aeMain+0x1d)[0x42eb0d]
      src/redis-server 127.0.0.1:21322(main+0x4c5)[0x42b145]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd3738a3830]
      src/redis-server 127.0.0.1:21322(_start+0x29)[0x42b409]
      
      Since we disconnect all the replicas and free the replication backlog in
      certain replication paths, and the code that will free the replication
      backlog expects that no replica is connected.
      
      However we still need to free the replicas asynchronously in certain
      cases, as documented in the top comment of disconnectSlaves().
      adc5df1b
    • antirez's avatar
      Fix #7306 less aggressively. · b407590c
      antirez authored
      Citing from the issue:
      
      btw I suggest we change this fix to something else:
      * We revert the fix.
      * We add a call that disconnects chained replicas in the place where we trim the replica (that is a master i this case) offset.
      This way we can avoid disconnections when there is no trimming of the backlog.
      
      Note that we now want to disconnect replicas asynchronously in
      disconnectSlaves(), because it's in general safer now that we can call
      it from freeClient(). Otherwise for instance the command:
      
          CLIENT KILL TYPE master
      
      May crash: clientCommand() starts running the linked of of clients,
      looking for clients to kill. However it finds the master, kills it
      calling freeClient(), but this in turn calls replicationCacheMaster()
      that may also call disconnectSlaves() now. So the linked list iterator
      of the clientCommand() will no longer be valid.
      b407590c
    • Qu Chen's avatar
      Disconnect chained replicas when the replica performs PSYNC with the master... · 42f5da5d
      Qu Chen authored
      Disconnect chained replicas when the replica performs PSYNC with the master always to avoid replication offset mismatch between master and chained replicas.
      42f5da5d
  10. 21 May, 2020 1 commit
  11. 16 May, 2020 1 commit
    • antirez's avatar
      Remove the client from CLOSE_ASAP list before caching the master. · 624742d9
      antirez authored
      This was broken in 1a7cd2c0: we identified a crash in the CI, what
      was happening before the fix should be like that:
      
      1. The client gets in the async free list.
      2. However freeClient() gets called again against the same client
         which is a master.
      3. The client arrived in freeClient() with the CLOSE_ASAP flag set.
      4. The master gets cached, but NOT removed from the CLOSE_ASAP linked
         list.
      5. The master client that was cached was immediately removed since it
         was still in the list.
      6. Redis accessed a freed cached master.
      
      This is how the crash looked like:
      
      === REDIS BUG REPORT START: Cut & paste starting from here ===
      1092:S 16 May 2020 11:44:09.731 # Redis 999.999.999 crashed by signal: 11
      1092:S 16 May 2020 11:44:09.731 # Crashed running the instruction at: 0x447e18
      1092:S 16 May 2020 11:44:09.731 # Accessing address: 0xffffffffffffffff
      1092:S 16 May 2020 11:44:09.731 # Failed assertion:  (:0)
      
      ------ STACK TRACE ------
      EIP:
      src/redis-server 127.0.0.1:21300(readQueryFromClient+0x48)[0x447e18]
      
      And the 0xffff address access likely comes from accessing an SDS that is
      set to NULL (we go -1 offset to read the header).
      624742d9
  12. 15 May, 2020 1 commit
    • antirez's avatar
      Cache master without checking of deferred close flags. · 1a7cd2c0
      antirez authored
      The context is issue #7205: since the introduction of threaded I/O we close
      clients asynchronously by default from readQueryFromClient(). So we
      should no longer prevent the caching of the master client, to later
      PSYNC incrementally, if such flags are set. However we also don't want
      the master client to be cached with such flags (would be closed
      immediately after being restored). And yet we want a way to understand
      if a master was closed because of a protocol error, and in that case
      prevent the caching.
      1a7cd2c0
  13. 27 Apr, 2020 1 commit
    • Oran Agra's avatar
      Keep track of meaningful replication offset in replicas too · 4447ddc8
      Oran Agra authored
      Now both master and replicas keep track of the last replication offset
      that contains meaningful data (ignoring the tailing pings), and both
      trim that tail from the replication backlog, and the offset with which
      they try to use for psync.
      
      the implication is that if someone missed some pings, or even have
      excessive pings that the promoted replica has, it'll still be able to
      psync (avoid full sync).
      
      the downside (which was already committed) is that replicas running old
      code may fail to psync, since the promoted replica trims pings form it's
      backlog.
      
      This commit adds a test that reproduces several cases of promotions and
      demotions with stale and non-stale pings
      
      Background:
      The mearningful offset on the master was added recently to solve a problem were
      the master is left all alone, injecting PINGs into it's backlog when no one is
      listening and then gets demoted and tries to replicate from a replica that didn't
      have any of the PINGs (or at least not the last ones).
      
      however, consider this case:
      master A has two replicas (B and C) replicating directly from it.
      there's no traffic at all, and also no network issues, just many pings in the
      tail of the backlog. now B gets promoted, A becomes a replica of B, and C
      remains a replica of A. when A gets demoted, it trims the pings from its
      backlog, and successfully replicate from B. however, C is still aware of
      these PINGs, when it'll disconnect and re-connect to A, it'll ask for something
      that's not in the backlog anymore (since A trimmed the tail of it's backlog),
      and be forced to do a full sync (something it didn't have to do before the
      meaningful offset fix).
      
      Besides that, the psync2 test was always failing randomly here and there, it
      turns out the reason were PINGs. Investigating it shows the following scenario:
      
      cycle 1: redis #1 is master, and all the rest are direct replicas of #1
      cycle 2: redis #2 is promoted to master, #1 is a replica of #2 and #3 is replica of #1
      now we see that when #1 is demoted it prints:
      17339:S 21 Apr 2020 11:16:38.523 * Using the meaningful offset 3929963 instead of 3929977 to exclude the final PINGs (14 bytes difference)
      17339:S 21 Apr 2020 11:16:39.391 * Trying a partial resynchronization (request e2b3f8817735fdfe5fa4626766daa938b61419e5:3929964).
      17339:S 21 Apr 2020 11:16:39.392 * Successful partial resynchronization with master.
      and when #3 connects to the demoted #2, #2 says:
      17339:S 21 Apr 2020 11:16:40.084 * Partial resynchronization not accepted: Requested offset for secondary ID was 3929978, but I can reply up to 3929964
      
      so the issue here is that the meaningful offset feature saved the day for the
      demoted master (since it needs to sync from a replica that didn't get the last
      ping), but it didn't help one of the other replicas which did get the last ping.
      4447ddc8
  14. 14 Apr, 2020 1 commit
  15. 28 Mar, 2020 1 commit
  16. 27 Mar, 2020 1 commit
  17. 25 Mar, 2020 1 commit
    • antirez's avatar
      PSYNC2: meaningful offset implemented. · 57fa355e
      antirez authored
      A very commonly signaled operational problem with Redis master-replicas
      sets is that, once the master becomes unavailable for some reason,
      especially because of network problems, many times it wont be able to
      perform a partial resynchronization with the new master, once it rejoins
      the partition, for the following reason:
      
      1. The master becomes isolated, however it keeps sending PINGs to the
      replicas. Such PINGs will never be received since the link connection is
      actually already severed.
      2. On the other side, one of the replicas will turn into the new master,
      setting its secondary replication ID offset to the one of the last
      command received from the old master: this offset will not include the
      PINGs sent by the master once the link was already disconnected.
      3. When the master rejoins the partion and is turned into a replica, its
      offset will be too advanced because of the PINGs, so a PSYNC will fail,
      and a full synchronization will be required.
      
      Related to issue #7002 and other discussion we had in the past around
      this problem.
      57fa355e
  18. 23 Mar, 2020 1 commit
  19. 04 Mar, 2020 4 commits
  20. 03 Mar, 2020 1 commit
  21. 25 Feb, 2020 1 commit
  22. 06 Feb, 2020 3 commits
  23. 31 Dec, 2019 1 commit
  24. 19 Dec, 2019 1 commit
  25. 19 Nov, 2019 1 commit
    • Johannes Truschnigg's avatar
      Use libsystemd's sd_notify for communicating redis status to systemd · 641c64ad
      Johannes Truschnigg authored
      Instead of replicating a subset of libsystemd's sd_notify(3) internally,
      use the dynamic library provided by systemd to communicate with the
      service manager.
      
      When systemd supervision was auto-detected or configured, communicate
      the actual server status (i.e. "Loading dataset", "Waiting for
      master<->replica sync") to systemd, instead of declaring readiness right
      after initializing the server process.
      641c64ad
  26. 29 Oct, 2019 1 commit
    • Oran Agra's avatar
      Modules hooks: complete missing hooks for the initial set of hooks · 51c3ff8d
      Oran Agra authored
      * replication hooks: role change, master link status, replica online/offline
      * persistence hooks: saving, loading, loading progress
      * misc hooks: cron loop, shutdown, module loaded/unloaded
      * change the way hooks test work, and add tests for all of the above
      
      startLoading() now gets flag indicating what is loaded.
      stopLoading() now gets an indication of success or failure.
      adding startSaving() and stopSaving() with similar args and role.
      51c3ff8d
  27. 10 Oct, 2019 1 commit
    • antirez's avatar
      Cluster: fix memory leak of cached master. · 747be463
      antirez authored
      This is what happened:
      
      1. Instance starts, is a slave in the cluster configuration, but
      actually server.masterhost is not set, so technically the instance
      is acting like a master.
      
      2. loadDataFromDisk() calls replicationCacheMasterUsingMyself() even if
      the instance is a master, in the case it is logically a slave and the
      cluster is enabled. So now we have a cached master even if the instance
      is practically configured as a master (from the POV of
      server.masterhost value and so forth).
      
      3. clusterCron() sees that the instance requires to replicate from its
      master, because logically it is a slave, so it calls
      replicationSetMaster() that will in turn call
      replicationCacheMasterUsingMyself(): before this commit, this call would
      overwrite the old cached master, creating a memory leak.
      747be463
  28. 07 Oct, 2019 2 commits
    • Yossi Gottlieb's avatar
      TLS: Configuration options. · 61733ded
      Yossi Gottlieb authored
      Add configuration options for TLS protocol versions, ciphers/cipher
      suites selection, etc.
      61733ded
    • Oran Agra's avatar
      diskless replication rdb transfer uses pipe, and writes to sockets form the parent process. · 5a477946
      Oran Agra authored
      misc:
      - handle SSL_has_pending by iterating though these in beforeSleep, and setting timeout of 0 to aeProcessEvents
      - fix issue with epoll signaling EPOLLHUP and EPOLLERR only to the write handlers. (needed to detect the rdb pipe was closed)
      - add key-load-delay config for testing
      - trim connShutdown which is no longer needed
      - rioFdsetWrite -> rioFdWrite - simplified since there's no longer need to write to multiple FDs
      - don't detect rdb child exited (don't call wait3) until we detect the pipe is closed
      - Cleanup bad optimization from rio.c, add another one
      5a477946