Conversation
... and shorten re-assignment interval.
Fixes some problems where connections wouldn't really reconnect or would spin up duplicate connections while the original one reconnected, causing "fun" conflicts. The solution here is "just let the underlying library handle it", more or less, and removes a lot of our manual handling attempts that were conflicting. It also consolidates a lot of the underlying connection handling.
The idea here is we try and "condense" job messages so we aren't flapping jobs. Like if we drop from the queue, a job is assigned, then cancelled, then re-assigned to us, we don't start it, stop it (sending a cancelation!), and then start the (now-canceled!) job, and instead pay attention to the current state that the coordinator has sent us, and act on that ONLY.
The problem here was timing out the async generator function which... resulted in breaking things after the timeout, meaning we never got further messages. Oops.
5 tasks
Owner
Author
|
I've been testing and running this in my environment (and that's how I actually caught the problem with timing out the async generators), and it has been running stable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix/improve the stability of Worker transport connection, especially on RabbitMQ. This introduces a bit of a refactor and resolves issues with connection handling and async generator issues.
Additionally lowers the default assignment timeout (to 10 Seconds) and "collapses" queued messages for the same job so we don't flap the state unnecessarily (and instead only treat the last message as the state we should action on).
Fixes #14
Type of change
Breaking Changes
Non-breaking, unless someone wants the longer assignment timeouts.
Checklist:
Code Quality
Testing
Architecture & Philosophy