Raft: Consensus for Rubyists

Transcript

1. Raft Consensus for Rubyists

2. @vanstee Big Nerd Ranch

3. Coffee Snob Scenario

4. Matt Amy Kim Dan

5. Matt Amy Kim Dan

6. Matt Amy Kim Dan Want to meet for coffee?

7. Matt Amy Kim Dan Totally! I’m in. zzzzz…

8. Matt Amy Kim Dan Ok, let’s meet at Octane.

9. Matt Amy Kim Dan Sounds good. I’ll be there. …

10. Matt Amy Kim Dan Who wants coffee? *wakes up

11. Matt Amy Kim Dan Who wants coffee?

12. Matt Amy Kim Dan Oh, we’re going to Octane.

13. Delicious Coffee Consensus

14. What did we just do?

15. • Leader election • State replication • Partition tolerance

16. Now write an algorithm.

17. Humans > Computers > Congress

18. Why Ruby

19. Multiple db servers? Multiple app servers? Multiple clients?

20. Multiple db servers? Distributed Multiple app servers? Multiple clients?

21. Multiple db servers? Distributed Multiple app servers? Distributed Multiple clients?

22. Multiple db servers? Distributed Multiple app servers? Distributed Multiple clients?

    Distributed

23. If you build webapps, you build distributed systems.

24. Raft

25. @ongardie Diego Ongaro John Ousterhout and

26. con·sen·sus /kənˈsensəs/ ! Agreeing upon state across distributed processes even

    in the presence of failures.

27. When should I use it?

28. • Distributed locks • Configuration • Background jobs

29. Why not Paxos?

30. Laying the Groundwork

31. {…} Log State Machine Consensus Module

32. Leader Follower Follower Follower

33. Leader Follower Follower Follower

34. Leader Follower Follower Follower

35. Follower

36. Leader Election Log Replication Safety

37. Follower Candidate Leader

38. Follower Candidate Leader

39. Follower Candidate Leader Times out, Starts election

40. Follower Candidate Leader Times out, Starts election Wins election

41. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election

42. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election Discovers current or new leader, Steps down

43. Follower Candidate Leader Times out, Starts election Times out, Restarts

    election Wins election Discovers new leader, Steps down Discovers current or new leader, Steps down

44. RequestVote Used by candidates to ask for votes during an

    election. Log information included for comparison. ! AppendEntries Used by leaders to tell followers which entries to replicate and commit. Also used as a heartbeat to remain leader. Follower Candidate Vote for me please. Follower Here are some entries for your log. Oh, and btw I’m still the leader. Leader

45. Term Higher numbers are used to determine leaders and check

    log entries. The term is incremented each time an election is started. Any command with an old term is ignored. Term 1 Term 2 Term 3 Election Normal Operation Split vote

46. Follower Follower Follower Follower

47. Candidate Follower Follower Follower Votes Times out, starts election

48. Candidate Follower Follower Follower Votes Votes for self

49. Candidate Follower Follower Follower Votes Sends out RequestVotes

50. Candidate Follower Follower Votes Responds with success

51. Leader Follower Votes Yay! I won! Follower

52. Leader Election Log Replication Safety

53. Log Entries Explained

54. SET X = 1 SET Y = 2 SET Z

    = 3 SET X = 4 SET X = 5 1 1 1 2 2 SET X = 1 SET Y = 2 SET Z = 3 SET X = 4 1 1 1 2 Leader Follower 1 2 3 4 5 6

55. { :entries => [{ 4 => 'SET X = 4'

    }], :term => 1, :prev_log_term => 1, :prev_log_index => 3, :leader_commit => 3, :leader_id => '192.168.1.101/7238' } APPEND ENTRIES EXAMPLE

56. SET X = 1 SET Y = 2 SET Z

    = 3 1 1 1 Before 1 1 1 1 After SET X = 1 SET Y = 2 SET Z = 3 SET X = 4 1 2 3 4 5 6 1 2 3 4 5 6

57. Log entries are always committed in the same order and

    are never uncommitted.

58. Happy Log Entry

59. Leader Follower Follower

60. Leader Follower Follower

61. Leader Follower Follower

62. Leader Follower Follower

63. Leader Follower Follower

64. Sad Log Entry

65. Leader Follower Follower

66. Leader Follower Follower

67. Leader Follower Follower

68. Leader Follower Follower

69. Leader Follower Leader

70. Leader Follower Leader

71. Leader Follower Leader

72. Leader Follower Leader

73. Leader Follower Leader

74. Follower Follower Follower Leader

75. Facepalm Log Entry

76. Leader Follower Follower

77. Leader Follower Follower

78. Leader Follower Follower

79. Leader Follower Follower

80. Leader Follower Follower

81. Leader Follower Follower

82. Leader Follower Leader

83. How do we guard against losing log entries?

84. Leader Election Log Replication Safety

85. Only cast votes for nodes with logs that contain at

    least as many entries as your own. 1

86. New leaders must commit a log entry form their new

    term before committing old entries. 2

87. Bonus Round

88. • Cluster changes • Log compaction • Client specifics

89. Why Ruby

90. None

91. Ruby is great for expressing complex problems.

92. Then why doesn’t the academic community love it?

93. • Community • Understandability • Learning Material

94. • Read Papers • Go To Conferences • Talk To

    People

95. Raft Paper Raft Implementations Raft Website ThinkDistributed Ricon Google Scholar

96. Question Time Make sure I repeat your questions.