Skip to content

refactor(fsdp)(4.3/5): family-declared input dtype policy at the model boundary - #103

Merged
Rockdu merged 3 commits into
radixark:mainfrom
Rockdu:feat/fsdp-input-dtype-policy
Aug 8, 2026
Merged

refactor(fsdp)(4.3/5): family-declared input dtype policy at the model boundary#103
Rockdu merged 3 commits into
radixark:mainfrom
Rockdu:feat/fsdp-input-dtype-policy

Conversation

@Rockdu

@Rockdu Rockdu commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto main now that #100 (refactor 4.2) is merged. Weight/param dtype control is not in this PR: the per-parameter dtype patch landed via #100; this is the input side only.


What

  • Each model family declares the dtypes of its model-boundary inputs (latents, cond, timestep) as an input_dtype_policy on its TrainPipelineConfig: a dtype name, "default" (the run's forward dtype), or None = pass the rollout dtype through. Default policy is all-None.
  • The trainer stops hard-casting forward inputs (latents/timesteps .to(forward_dtype) and cast_cond_to_dtype removed; FSDP cast_forward_inputs=False) and instead applies the family policy once at the boundary, then runs the DiT forward under torch.autocast(forward_dtype).
  • LTX opts in (latents/cond = "default", timestep passthrough) and its inner per-forward autocast folds into the trainer's ambient one.

Why

Boundary input dtypes are what element-wise ops see before any weight is involved, so they must match what the family's sglang-d pipeline feeds the DiT for train/rollout log-prob alignment — and that is per-family knowledge, not a global cast. With the boundary owned by the policy, compute is owned by autocast; the grad-ckpt buffer force-cast in apply_fsdp2 is dropped because autocast keeps recompute dtypes consistent without rewriting fp32 buffers (e.g. RoPE tables).

Files

  • miles/backends/fsdp_utils/input_dtype_policy.py — the policy keys and apply_input_dtype_policy
  • miles/backends/fsdp_utils/actor.py — boundary policy application + ambient autocast; cast_forward_inputs=False; buffer force-cast removed
  • miles/backends/fsdp_utils/configs/train_pipeline_config.pyinput_dtype_policy field (default passthrough)
  • miles/backends/fsdp_utils/configs/ltx.py — LTX policy; inner autocast removed
  • miles/backends/fsdp_utils/loss_hub/{flow_grpo,nft,utils}.py — cond-cast removal (utils.py deleted, it only held the cast)
  • tests/fast/backends/fsdp_utils/test_input_dtype_policy.py — 4 CPU tests (passthrough default, family override, float-only casting, unknown-key rejection), registered to stage-a-cpu
  • tests/ci/fixtures/e2e_standards/test_sd3_ocr_grpo_2xGPU.json — re-recorded: autocast + boundary-cast changes shift SD3 numerics

Checklist

  • pre-commit run --all-files passes
  • Added/updated tests for new behaviour
  • pytest -x is green — not run locally (no local torch); the same 4 policy tests were green on this PR's previous CI run, and stage-a-cpu re-runs them here
  • If launch flags changed, python3 train.py --help still parses — no flags changed
  • If a public flag was added, it appears in the CLI reference docs — no flag added
  • If an example was added, it has a real walkthrough — no example added

@Rockdu Rockdu changed the title refactor(fsdp)(4.0/5): family-declared input dtype policy at the model boundary refactor(fsdp)(4.3/5): family-declared input dtype policy at the model boundary Aug 7, 2026
@Rockdu
Rockdu force-pushed the feat/fsdp-input-dtype-policy branch 3 times, most recently from 0bac0d2 to 05c6634 Compare August 8, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant