fix(diffusion): re-inject clean first-frame image during OmniDreams context-noise KV cache write - #2
fix(diffusion): re-inject clean first-frame image during OmniDreams context-noise KV cache write#2Cerdore wants to merge 6 commits into
Conversation
Add OmniDreams pipeline to multimodal_gen, supporting:
- Flat checkpoint loading (570-key .pt, non-diffusers layout) with
post-load parameter fusion matching FlashDreams conventions
- Autoregressive rollout with BlockKVCache (sink + rolling window)
and per-chunk KV-cache lifecycle
- 3D NeoX RoPE (44:42:42) with shift_t for chunk positioning
- 2-step flow-match scheduler (sigmas {1.0, 0.8036, 0.0})
- Text conditioning via full_concat 100352 embedding
(Cosmos-Reason1-7B) with explicit attention mask
- Per-chunk VAE latent concatenation with single-pass Wan 2.1 decode
for temporal continuity
- HDMap/trajectory conditioning pipeline (per-chunk VAE-encode)
- Tensor parallelism via ColumnParallelLinear/RowParallelLinear
- Precomputed cross-attention K/V caching
- HTTP API fields for hdmap_path and num_views
- 43/43 CPU unit tests covering component construction, RoPE,
BlockKVCache, scheduler, DiT forward, denoising stage, registry,
and regression guards
- GPU CI coverage with consistency thresholds and perf baselines
Relates to sgl-project#27214
… HD-map conditioning Consolidate pre-processing (text encode, i2v VAE, HD-map per-chunk encoding) into OmniDreamsBeforeDenoisingStage. Implement full autoregressive rollout in OmniDreamsDenoisingStage: per-block BlockKVCache lifecycle, 3D NeoX RoPE shift_t, 2-step self-forcing denoise (sample + context-noise re-forward), frame-0 i2v pinning, and per-chunk HD-map indexing. Add comprehensive unit tests covering RoPE correctness, KV-cache fill/roll/steady-state, scheduler 2-step sigmas, Cosmos-Reason1 100352-dim text embedding, tiny DiT end-to-end forward, AR unconditioned/i2v/multi-chunk rollout, HD-map per-frame slicing, and reference image preprocessing. Constraint: 2-step distilled model trades temporal smoothness for real-time speed; blur is expected behavior from Self-Forcing + DMD distillation, not a bug. Confidence: high Scope-risk: moderate (AR loop is new; GPU validation still needed) Not-tested: end-to-end generation on full OmniDreams 2B checkpoint; HD-map VAE numerics on GPU (flagged with TODO) Co-Authored-By: Claude <noreply@anthropic.com>
…ontext-noise KV cache write At AR step 0, the context-noise forward (finalize_kv_cache) must store CLEAN K/V for frame 0, matching FlashDreams _maybe_inject_image() inside _predict_branch which fires during finalize_kv_cache -> predict_flow. Without this re-injection, frame-0 KV cache entries are noise-corrupted at sigma ~0.13, causing a distribution shift that accumulates into progressively blurry video across AR chunks.
|
以下是 AgainstEntropy 6 条意见在当前代码上的状态: 意见 |
Problem
The SGLang OmniDreams port produces blurry video where only the first frame
is sharp and every generated frame after it is blurry. The VAE decoder is
fine -- the bug is in the autoregressive denoising rollout.
Root Cause
At AR step 0, the context-noise forward (the final self.transformer() call
that writes the authoritative K/V into the rolling KV cache) feeds a
noise-corrupted first-frame latent to the network. After denoising ->
postprocess_clean_latent (which re-injects the clean VAE-encoded first-frame
image at frame 0), the code calls scheduler.add_noise(clean, 128) which
corrupts ALL frames at sigma ~0.13. This noisy latent is then fed directly
to the transformer for the KV-cache write -- the first frame's K/V in the
cache are noise-corrupted.
In the FlashDreams reference, _maybe_inject_image() fires INSIDE
_predict_branch during finalize_kv_cache -> predict_flow, re-injecting the
clean first-frame VAE latent AFTER add_noise but BEFORE the network forward
that writes the final KV cache. This means FlashDreams stores CLEAN frame-0
K/V, while SGLang stores NOISY frame-0 K/V.
When subsequent AR chunks attend to the first frame's K/V, they see
corrupted keys/values -- a distribution shift that the model was not
trained to handle, which accumulates into progressively blurry video
across AR chunks.
Fix
One if pin: block (2 lines) inserted after scheduler.add_noise() and before
self.transformer() in the context-noise KV cache write path.
File: python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/omnidreams.py
Diff:
For AR step >= 1, pin is False, so the new block is a no-op -- matching
FlashDreams where _maybe_inject_image short-circuits on
cache.autoregressive_index != 0.
CI States
Latest PR Test (Base): ❌ Run #27461916594
Latest PR Test (Extra): ❌ Run #27461916363