Fix load_snapshot() error handling bug (#162)
Consider the snapshot delivery scenario: - Follower receives the last chunk, calls store_snapshot_chunk() succesfully and updates its snapshot_recv_offset. Later it calls load_snapshot() but it fails. - Follower returns failure response. In the response, "offset" will be zero. So, follower requests leader to start snapshot delivery from the beginning. - Leader sends the first chunk. Follower rejects the message as its snapshot_recv_offset is not zero (Actually, follower has stored all the bytes, snapshot_recv_offset is equal to snapshot size). Follower returns this offset in the response. - Leader calls get_snapshot_chunk() callback for the received offset from the follower. As there are no more bytes to send, get_snapshot_chunk() returns RAFT_ERR_DONE and leader skips sending snapshot req. This bug causes leader to stop sending appendreq/snapshotreq to the follower until leader takes another snapshot. Follower starts an election as it doesn't hear from the leader but other nodes won't vote for it as the rest of the cluster operates fine. Solution is to consider store_snapshot_chunk() and load_snapshot() as an atomic unit. So, if load_snapshot() fails, we return the response as if like we didn't store the last chunk. Leader will send the same chunk again and we can retry the operation.
Please register or sign in to comment