• Ozan Tezcan's avatar
    Fix load_snapshot() error handling bug (#162) · 9a0c6c0e
    Ozan Tezcan authored
    Consider the snapshot delivery scenario:
    
    - Follower receives the last chunk, calls store_snapshot_chunk() 
    succesfully and updates its snapshot_recv_offset. Later it calls
    load_snapshot() but it fails. 
    
    - Follower returns failure response. In the response, "offset" will be
    zero. So, follower requests leader to start snapshot delivery from the
    beginning.
    
    - Leader sends the first chunk. Follower rejects the message as its 
    snapshot_recv_offset is not zero (Actually, follower has stored all the
    bytes, snapshot_recv_offset is equal to snapshot size). Follower returns
    this offset in the response.
    
    - Leader calls get_snapshot_chunk() callback for the received offset from
    the follower. As there are no more bytes to send, get_snapshot_chunk()
    returns RAFT_ERR_DONE and leader skips sending snapshot req.
    
    This bug causes leader to stop sending appendreq/snapshotreq to the 
    follower until leader takes another snapshot. Follower starts an 
    election as it doesn't hear from the leader but other nodes won't vote
    for it as the rest of the cluster operates fine.
    
    Solution is to consider store_snapshot_chunk() and load_snapshot() as 
    an atomic unit. So, if load_snapshot() fails, we return the response as
    if like we didn't store the last chunk. Leader will send the same chunk
    again and we can retry the operation.
    9a0c6c0e
raft_server.c 64.6 KB