refactor(fsdp)(4.3/5): family-declared input dtype policy at the model boundary - #103
Merged
Merged
Conversation
Rockdu
force-pushed
the
feat/fsdp-input-dtype-policy
branch
3 times, most recently
from
August 8, 2026 00:31
0bac0d2 to
05c6634
Compare
Rockdu
force-pushed
the
feat/fsdp-input-dtype-policy
branch
from
August 8, 2026 00:32
05c6634 to
ec23b50
Compare
4 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
latents,cond,timestep) as aninput_dtype_policyon itsTrainPipelineConfig: a dtype name,"default"(the run's forward dtype), orNone= pass the rollout dtype through. Default policy is all-None.latents/timesteps .to(forward_dtype)andcast_cond_to_dtyperemoved; FSDPcast_forward_inputs=False) and instead applies the family policy once at the boundary, then runs the DiT forward undertorch.autocast(forward_dtype).latents/cond = "default",timesteppassthrough) and its inner per-forward autocast folds into the trainer's ambient one.Why
Boundary input dtypes are what element-wise ops see before any weight is involved, so they must match what the family's sglang-d pipeline feeds the DiT for train/rollout log-prob alignment — and that is per-family knowledge, not a global cast. With the boundary owned by the policy, compute is owned by autocast; the grad-ckpt buffer force-cast in
apply_fsdp2is dropped because autocast keeps recompute dtypes consistent without rewriting fp32 buffers (e.g. RoPE tables).Files
miles/backends/fsdp_utils/input_dtype_policy.py— the policy keys andapply_input_dtype_policymiles/backends/fsdp_utils/actor.py— boundary policy application + ambient autocast;cast_forward_inputs=False; buffer force-cast removedmiles/backends/fsdp_utils/configs/train_pipeline_config.py—input_dtype_policyfield (default passthrough)miles/backends/fsdp_utils/configs/ltx.py— LTX policy; inner autocast removedmiles/backends/fsdp_utils/loss_hub/{flow_grpo,nft,utils}.py— cond-cast removal (utils.pydeleted, it only held the cast)tests/fast/backends/fsdp_utils/test_input_dtype_policy.py— 4 CPU tests (passthrough default, family override, float-only casting, unknown-key rejection), registered tostage-a-cputests/ci/fixtures/e2e_standards/test_sd3_ocr_grpo_2xGPU.json— re-recorded: autocast + boundary-cast changes shift SD3 numericsChecklist
pre-commit run --all-filespassespytest -xis green — not run locally (no local torch); the same 4 policy tests were green on this PR's previous CI run, and stage-a-cpu re-runs them herepython3 train.py --helpstill parses — no flags changed