[flytekit]: fix pod_template serialization for dynamic task node overrides#38
Open
[flytekit]: fix pod_template serialization for dynamic task node overrides#38
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
jld-adriano
approved these changes
Feb 19, 2026
…fast_serialize The pod_spec in TaskNodeOverrides was only serialized when should_fast_serialize() was True. This meant dynamic tasks with pod_template overrides (via with_overrides) would produce an empty pod_spec in the DynamicJobSpec when fast_serialize was disabled, causing the propeller to fail to schedule the child task (UNKNOWN status). Aligns with upstream flytekit behavior: always serialize pod_spec when a pod_template override exists; only gate set_command_fn behind should_fast_serialize(). Co-Authored-By: benchan@exa.ai <ben@vervious.com>
91f78e0 to
793ca45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
When a dynamic task uses
with_overrides(pod_template=...), the pod spec in theTaskNodeOverrideswas only serialized whenshould_fast_serialize()returned True. During dynamic task compilation at runtime,should_fast_serialize()is typically False, sooverride_pod_specremained{}(empty dict). However, theTaskNodeOverridesstill constructed aPodTemplate(pod_spec={}, ...)becauseentity._pod_templateis checked separately downstream. This resulted in the propeller receiving a node with an empty pod spec override, causing the child task to stay in UNKNOWN status.This was observed in execution byc-812vfva9fvq7 where a post-training dynamic task's child showed UNKNOWN status indefinitely.
The fork had diverged from upstream flytekit here — upstream always calls
_serialize_pod_specwhen a pod_template exists, and only gatesset_command_fnbehindshould_fast_serialize().What changes were proposed in this pull request?
Moved
_serialize_pod_specoutside theshould_fast_serialize()gate so it's called wheneverentity._pod_template is not None, matching upstream behavior. Theset_command_fncall (which rewrites the entrypoint for fast registration) remains gated behindshould_fast_serialize()with the existingContainerTaskguard.Before:
After:
How was this patch tested?
Human review checklist
_serialize_pod_spec+_get_containerproduce valid output whenset_command_fnhas NOT been called (non-fast-serialize path). This is the main behavioral change.ContainerTaskisinstance check) is preserved in the new structure.Check all the applicable boxes
Link to Devin run: https://app.devin.ai/sessions/2d5b0a201d6d472db332fd9f20485019
Requested by: @Vervious