Dynamic (pad-to-batch-max) padding instead of fixed llm_input_len

[TonyChen06]

Summary

Pad each batch to its longest member (in the collate) instead of padding every
sample up to --llm_input_len. llm_input_len becomes a truncation ceiling only,
so short batches stop spending compute on padding they don't need.

Benchmark

PTB-XL ECG-QA, 132 examples,
identical config
llm_input_len | fixed padding (before) | dynamic padding (this PR) | speedup
--------------+------------------------+---------------------------+--------
512 | 11 s (3.57 it/s) | 6 s (5.36 it/s) | 1.8x
1024 | 19 s (2.12 it/s) | 6 s (5.36 it/s) | 3.2x
2048 | 37 s (1.10 it/s) | 6 s (5.39 it/s) | 6.2x

Dynamic-padding runtime is flat across caps (batches pad to the real content
length).

What changed

• Representations (signal / rgb / stacked_signal / symbolic): trunc_pad_input
  no longer pads — it returns the unpadded (only-truncated) sequence, and the
  per-sample length asserts drop the == llm_input_len term.
• Base.pad_to_batch: left-pads input_ids / attention_mask / labels to the batch
  max and offsets the stored signal-token indices by the pad amount, so encoder-
  injection positions track the shift.
• collate_fn: pads to the batch max right before the existing structure-assert
  and default_collate — the rest of the pipeline is unchanged.
• torch.compile(dynamic=True): one graph for the now batch-varying seq length.

Padding side is unchanged (still left-padding), so loss/generation semantics are
identical. Validated across all four representations, single-GPU and 4-GPU DDP,
and torch.compile.

⚠️ Warning

With fixed padding, every batch was max-length, so any length-related OOM shows up on step ~1.
With dynamic padding, a long batch may not appear until mid-epoch, so a run
can train happily for a while and then OOM when a long batch finally lands.

1-2 follow-up PR will add possible solutions to this problem, but they are quite subjective so they are separate from this feature's PR.

🤖 Generated with Claude Code

[TonyChen06]

Follow up fixes done! They are separate PRs and also include the changes here.

[willxxy]

Overall Comments:

Similar to the previous PR #72 , these kind of more fundamental changes would greatly benefit from the changer to know exactly everything on whats happening. I would rather not make these kind of changes before knowing exactly what it is.
I know empirically for modern llm training people do pad to batch max, but i have not read too much on this kind of stuff. Do you know about this area confidently? what are the pros, what are cons and how will it affect our projects?
You don't have to list them out here but lets discuss more in person about it.