1. 25 Oct, 2022 2 commits
    • Ozan Tezcan's avatar
      Refactor higher term handling (#149) · 2ce5363f
      Ozan Tezcan authored
      This PR refactors code regarding `term` handling inside 
      `raft_recv_appendentries()` and `raft_recv_snapshot()` without 
      introducing any logic changes. 
      
      - Deleted `raft_receive_term()` function:
          This function sets the node as follower in certain cases. Although,
          it misses the pre-candidate state.
           
          We already set the node as follower inside `raft_accept_leader()`.
          (With the correct check that also handles pre-candidate state). 
      
      - Moved `raft_reset_transfer_leader()` into `raft_accept_leader()`. 
          Leader transfer is completed when we accept a leader. So, it makes
          sense to move it into `raft_accept_leader()`.
          
      - Added some tests to verify node becomes follower even on a
           pre-candidate state when a request is received with same or
           higher term.
      2ce5363f
    • Ozan Tezcan's avatar
      Refactor raft_recv_appendentries() (#148) · 7be57f36
      Ozan Tezcan authored
      Refactored raft_recv_appendentries() just to simplify the flow.
      
      This PR should not introduce any behavioral change. Only exception is
      previously, some tests were manually creating an append request with
      prev_log_idx:0 and prev_log_term:1. This is not possible and it
      cannot happen without manually creating append requests. Term of the
      entry at index 0 cannot be 1. (it is zero by definition). Fixed these tests
      as part of this PR.
      7be57f36
  2. 03 Oct, 2022 1 commit
  3. 19 Sep, 2022 1 commit
    • Ozan Tezcan's avatar
      Handle unnecessary snapshots better (#143) · 85ae16e0
      Ozan Tezcan authored
      If a node receives a snapshot with snapshot_last_idx, it checks if 
      it already has the snapshot and informs leader accordingly.
      
      The problem is the receiver node may not have a snapshot but log 
      entries more than snapshot_last_idx:
      
      e.g
      Leader: snapshot_last_idx : 50
      Follower: snapshot_last_idx:0, last log entry index: 100
      
      If the leader tries to send snapshot to the follower 
      (with snapshot last index:50), currently, follower accepts 
      the snapshot. As follower already has entries upto index 100, snapshot
      is unnecessary. After delivery is complete, to load the snapshot, 
      follower will call raft_begin_load_snapshot(). This function realizes
      the snapshot was unnecessary and it fails. It is harmless but snapshot
      delivery was unnecessary.
      
      This PR prevents this issue earlier by checking last log index instead
       of snapshot last index.
      
      Also, added more virtraft tests with auto-flush on. 
      (This is a different pattern, it might help with detecting these issues)
      85ae16e0
  4. 12 Sep, 2022 1 commit
    • Ozan Tezcan's avatar
      Handle response error cases gracefully (#137) · 8db9930c
      Ozan Tezcan authored
      A node can receive responses from followers who are not
      aware of they have been removed from the cluster configuration.
      In this case, recv_ family functions can be called without 
      a node structure. We should ignore these messages.
      
      We should also ignore snapshot and appendentries responses
      if we are not leader anymore. Returning an error code for 
      this case does not make sense because application cannot 
      do anything but ignore the error code. 
      
      This PR changes return value to zero for these cases. 
      8db9930c
  5. 05 Sep, 2022 1 commit
  6. 04 Sep, 2022 1 commit
  7. 30 Aug, 2022 1 commit
    • Ozan Tezcan's avatar
      Refactor debug logging (#127) · e79d3b6a
      Ozan Tezcan authored
      Refactor debug logging and change log callback
      
      e.g:
      19 --> 13, sent snapshot_resp id:27712, t:24, s:1, o:4096, lc:0
      14 --> 17, sent requestvote_req pv:1, t:25, ci:14, lli:6848, llt:23
      14 <-- 17, recv requestvote_resp pv:1, t:25, rt:24, vg:0
      
      Breaking changes: 
      
      raft_log_f() callback will not pass node_id_t anymore.
      Sender and receiver will be part of the log line as you can see above.
      e79d3b6a
  8. 29 Aug, 2022 2 commits
    • Ozan Tezcan's avatar
      Simplify entry removal (#130) · c0270e61
      Ozan Tezcan authored
      Simplied pop callback.
      
      Previously, libraft was passing a callback(raft_entry_notify_f) to the pop callback.
      The application was supposed to call that callback for each removed entry.
      
      Libraft can loop over the entries and call the required functions itself.
      So, we can delete raft_entry_notify_f from the pop callback just to
      simplify API.
      
      In addition to that, this PR removes raft_pop_entry() function which doesn't have
      a use-case, as far as I can see.
      We can use raft_delete_entry_from_idx(me, raft_get_current_idx()) instead if required.
      c0270e61
    • Ozan Tezcan's avatar
      Add raft_get_last_applied_term() (#131) · 2279a960
      Ozan Tezcan authored
      Add raft_get_last_applied_term()
      2279a960
  9. 25 Aug, 2022 1 commit
    • Ozan Tezcan's avatar
      Add support to rebuild configuration after a restart (#125) · 9c09a6cc
      Ozan Tezcan authored
      Configuration changes are special in Raft. They take effect 
      whenever the related log entries are appended. On a restart, 
      we should go over the log entries and rebuild the configuration.
      
      Adding raft_restore_log() function for this purpose.
      
      Overall, on a restart, application should do:
      
      Load the snapshot and call raft_restore_snapshot()
      Load the log entries and call raft_restore_log()
      Read the metadata file and call raft_restore_metadata().
      9c09a6cc
  10. 23 Aug, 2022 1 commit
    • Ozan Tezcan's avatar
      Add raft_persist_metadata callback (#118) · 87c205e0
      Ozan Tezcan authored
      Combine raft_persist_vote_f and raft_persist_term_f into 
      a single callback: raft_persist_metadata_f.
      
      Users should persist vote and term into a single metadata file.
      
      After a restart, application should call raft_restore_metadata() 
      to restore vote and term
      87c205e0
  11. 22 Aug, 2022 1 commit
    • Ozan Tezcan's avatar
      Improve snapshot support (#121) · 7d66a156
      Ozan Tezcan authored
      - raft_load_snapshot_f(): Changed order of arguments term and index
      to be inline with raft_begin_load_snapshot().
      
      - Refactored raft_begin_load_snapshot(). Now, inside this function, 
      current server's node flags are cleared. Previously, application had to
      take care of it. e.g Application had to clear inactive flag on a snapshot.
      
      - Added raft_restore_snapshot() to let libraft know that application has
      loaded a snapshot.
      7d66a156
  12. 08 Aug, 2022 1 commit
  13. 21 Jul, 2022 1 commit
  14. 07 Jul, 2022 2 commits
  15. 30 Jun, 2022 1 commit
  16. 19 Jun, 2022 1 commit
  17. 19 May, 2022 1 commit
  18. 15 May, 2022 1 commit
  19. 02 May, 2022 1 commit
  20. 26 Apr, 2022 1 commit
  21. 06 Apr, 2022 1 commit
  22. 04 Apr, 2022 1 commit
  23. 03 Apr, 2022 1 commit
  24. 30 Mar, 2022 2 commits
  25. 24 Mar, 2022 1 commit
  26. 21 Mar, 2022 1 commit
  27. 13 Mar, 2022 1 commit
  28. 07 Feb, 2022 1 commit
  29. 17 Nov, 2021 1 commit
  30. 16 Nov, 2021 1 commit
    • Yossi Gottlieb's avatar
      Properly get coverage from all tests. (#70) · ada3d6a0
      Yossi Gottlieb authored
      * Add COVERAGE=1 Makefile option to build libraft with coverage.
      * Refactor CFFI module to use the existing shared object in an
        out-of-line configuration, so there's no need to build and re-build
        it.
      * Makefile adjustments to build the module CFFI as needed.
      * Remove raft_get_snapshot_entry_idx, a prototype for an undefined
        function that breaks CFFI.
      * Add a gcov Makefile target to fetch both unit tests and integration
        test coverage in one shot.
      ada3d6a0
  31. 15 Nov, 2021 1 commit
  32. 11 Nov, 2021 1 commit
  33. 03 Nov, 2021 1 commit
    • Shaya Potter's avatar
      set next_idx on each succesful raft_send_appendentries call (#65) · 02bdf818
      Shaya Potter authored
      this is to avoid sending duplicate entries each time raft_send_appendentries is called, just because they haven't been acked yet.
      
      it will still be reset backwards upon the leader receiving a raft_recv_appendentries_response() with a failure mode
      
      + a test for next_idx on append entry, which changes a number of tests which checked for this
      
      in order to do this, I had to modify the existing mock callback to return 0 instead of 1.  this needs to be verified
      
      + fixes for raft tests that this change exposed
      
      1) when we issues a sequence of entries, each was given an increasing term, so had entries whose term was higher than leaders term
      
      so had to fix that
      
      2) by setting the term of appendentries_repsonse to not be equal to leaders term (because of fix above), we now fail on receiving them
      
      do all that by not allowing raft_append_entry to append an entry that break raft semantics (i.e. term of entry > log term), but this breaks many tests that don't set term, so set them
      02bdf818
  34. 07 Oct, 2021 1 commit
    • Shaya Potter's avatar
      redo/tweak transfer leader in libraft (#61) · 63b0b1c1
      Shaya Potter authored
      this started off as a small tweak and grew a bit
      
      1) new callback for transfer leader result, along with its own enum of states
      
      2) change/remove raft_reset_transfer_leader() calls
      
      only call in places where there's a new leader or timeout (i.e. failed to transfer within period).
      
      so, we only reset in 3 places 1) recv_appendentries 2) raft_periodic (timeout) 3) raft_become_leader
      
      2a) raft_become_leader also got refactored a little to only issue the normal cb.notify_state_event() callback after it has set its state to be leader (as before, it could fail after the callback was issued)
      
      2b) in raft_periodic the raft_reset_transfer_leader() is pulled out the "if LEADER" block, as will lose leader when a vote is issued to it, we still want it to timeout if leadership isn't gained
      
      3) raft_reset_transfer_leader() does the logic for determining success/failure now, but becaue timeout is its own result which can't be determined by just looking at leader/desired, add a new flag to it to note timeout state
      
      4) leader stickiness in requestvote is only tested for prevote (as in a timeout now case, we want to drop the timeout now flag after prevotes are sent).  The idea is that we will only get to actual election if prevote passes.
      
      4) a bunch of new tests
      
      * fix issue with timeout flag to reset_transfer_leader
      
      also fix test to catch the original error
      
      * libraft/redisraft changes needed for e2e leader transfer
      
      * make leader stickiness only for prevote
      63b0b1c1
  35. 23 Sep, 2021 2 commits
    • Shaya Potter's avatar
      Adding support for TimeoutNow RPC (#7) · 575335c3
      Shaya Potter authored
      this is the small modifications needed for raftlib to support timeout now rpc
      
      1) add a flag to requestvote type to tell other nodes its overriding normal chec
      2) new send_timeoutnow callback to send a timeout now rpc at right time
      3) call said callback is we've marked the node as timeout now and in raft_appendentries_response raft_get_current_idx(me_) == r->current_idx (i.e. response says the node's idx is up to date with leader)
      4) add helper functions to set/get/reset timeout flag on node structs for targeted node.
      
      how it would be used by client
      
      1) client of libraft will call raft_transfer_leader(raft_server_t* me_, raft_node_id_t node_id) to target the node they want to transfer to.
      
      2) client of libraft will provide the send_timeoutnow() callback for the actual sending of the rpc
      
      3) client of libraft will modify their existing notify_state_event() callback to observe the result of the timeout now operation.
      
      if it receives a RAFT_STATE_LEADERSHIP_TRANSFER_FAILED, then the transfer failed
      
      if it received a RAFT_STATE_FOLLOWER, then it inspects for the actual leader to see if its the expected one and can return that it was transferred to expected node or not
      
      if it received a RAFT_STATE_CANDIDATE, then the transfer also failed, as the targeted node sent a request vote which removed leadership, but wasn't able to win the election.
      
      With PreVote, this latter case should be very rare.
      575335c3
    • Ozan Tezcan's avatar
      Define separate callback function for node id retrieval (#55) · ce8b0f02
      Ozan Tezcan authored
      Define separate callback function for node id retrieval
      ce8b0f02