Track and report non-responsive followers when quorum rounds age-out#17
Open
GeoffreyPS wants to merge 2 commits intochassisframework:mainfrom
Open
Conversation
GeoffreyPS
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
This pull request includes minor changes to the QuorumStatus module to allow Craft to report on nodes that did not send heartbeat responses by the time a quorum round stops being tracked.
With these changes, craft can now publish telemetry events for each round that ages out for all followers that did not respond in time. The new telemetry event is
[:craft, :quorum, :miss]Effectively, this gives us information on non-responsive followers without having to wait for such a follower to eventually send us the heartbeat (when we ordinarily publish the
:missed_deadlinetelemetry event).Extra Info
Reviewing notes
Note: There are a few changes here that are just formatting. I noticed this early enough to be able to run the formatter without any new work and create a formatting change as a separate commit. That way, it can be easier to view the diff of what really changed. I may have a formatter mismatch and can check on getting local environment matching if this is not desirable.
Additional work will be needed on either this branch or Add observability docs #16, depending on the order of their merging (assuming changes are accepted).
I do not know the best place to insert any new tests to exercise this behavior. I would welcome such suggestion.