• antirez's avatar
    Sentinel: only demote old master into slave under certain conditions. · fb0d08e3
    antirez authored
    We used to always turn a master into a slave if the DEMOTE flag was set,
    as this was a resurrecting master instance.
    
    However the following race condition is possible for a Sentinel that
    got partitioned or internal issues (tilt mode), and was not able to
    refresh the state in the meantime:
    
    1) Sentinel X is running, master is instance "A".
    3) "A" fails, sentinels will promote slave "B" as master.
    2) Sentinel X goes down because of a network partition.
    4) "A" returns available, Sentinels will demote it as a slave.
    5) "B" fails, other Sentinels will promote slave "A" as master.
    6) At this point Sentinel X comes back.
    
    When "X" comes back he thinks that:
    
    "B" is the master.
    "A" is the slave to demote.
    
    We want to avoid that Sentinel "X" will demote "A" into a slave.
    We also want that Sentinel "X" will detect that the conditions changed
    and will reconfigure itself to monitor the right master.
    
    There are two main ways for the Sentinel to reconfigure itself after
    this event:
    
    1) If "B" is reachable and already configured as a slave by other
    sentinels, "X" will perform a redirection to "A".
    2) If there are not the conditions to demote "A", the fact that "A"
    reports to be a master will trigger a failover detection in "X", that
    will end into a reconfiguraiton to monitor "A".
    
    However if the Sentinel was not reachable, its state may not be updated,
    so in case it titled, or was partiitoned from the master instance of the
    slave to demote, the new implementation waits some time (enough to
    guarantee we can detect the new INFO, and new DOWN conditions).
    
    If after some time still there are not the right condiitons to demote
    the instance, the DEMOTE flag is cleared.
    fb0d08e3
sentinel.c 119 KB