Add seeded Text LLM sampling

[JPPhoto]

Summary

Adds controlled seeded sampling to the Text LLM and Text LLM with System Prompt Preset nodes.

• Exposes a validated seed input and bumps both nodes to version 1.1.0.
• Uses invocation-local generators to prevent concurrent requests from interfering with each other's RNG state.
• Preserves varied Expand Prompt results by selecting a fresh seed per request.
• Documents seed behavior and reproducibility boundaries.

Related Issues / Discussions

Discord.

QA Instructions

1. Run:
   pytest -n auto --no-cov tests/backend/text_llm tests/app/invocations/test_text_llm_with_preset.py
2. Confirm all tests pass.
3. Run the same Text LLM node twice with identical inputs and seed; confirm identical output on the same platform.
4. Change the seed; confirm sampling can produce different output.
5. Run concurrent seeded CPU sampling; confirm each request matches its isolated sequence.

Merge Plan

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[Pfannkuchensack]

PR #9451 — Add seeded Text LLM sampling

• The tests do not prove the feature. Two independent one-line mutations destroy seeded sampling and leave all 30 tests green:
  • removing the self._generators cache (invokeai/backend/text_llm_pipeline.py:40-44) so the RNG never advances — measured effect: torch.multinomial returns the same value 10/10 times, i.e. a real model emits one repeated token;
  • moving with _SeededMultinomialMode(seed): (:127) out of the worker closure — TorchFunctionMode is thread-local, so the mode never covers generate and the seed does nothing.
    Cause: the RNG tests call torch.multinomial by hand inside the mode and never go through run(); everything that does go through run() uses a MagicMock model that never samples. An end-to-end test with a tiny randomly-initialised causal LM costs 0.18 s for three CPU runs and kills both mutations.
• Runtime cost: +19.4% on every text-LLM invocation. Measured on real Qwen2.5-1.5B-Instruct weights (RTX 4090, bf16, 128 tokens, median of 5, non-overlapping ranges): 19.81 → 23.66 ms/token, +3.85 ms/token, from 1931 intercepted torch calls per token. That is ~+1.16 s at the node default max_tokens=300 and ~+6 s at the 2048 cap, on a path where the user watches tokens stream. The mode wraps the whole forward pass, not just sampling — narrowing it to the sampling step (a custom LogitsProcessor with explicit Temperature/TopP warpers) would be O(1) per token. If the cost is accepted, put it in the PR description.
• Saved workflows silently become deterministic. The 1.0.0 → 1.1.0 bump makes validateWorkflow auto-run updateNode, whose defaultsDeep fills the new seed with the template default 0. A workflow that previously varied on each run now returns byte-identical text forever, with no signal to the user and no way back except manually wiring a rand_int. (Masked inside one server process by the node cache; visible across restarts.) Add a vitest case pinning the updateNode outcome for a stored text_llm node.
• func is torch.multinomial misses the method form. p.multinomial(...) bypasses seeding entirely (verified: reproducible via the function form, not via the method form). Unreachable today — all four transformers sampling paths were checked and ruled out — but the method form already exists elsewhere in the same transformers version, so an upstream refactor would ship broken reproducibility with a fully green suite.
• Expand Prompt cannot reproduce anything. invokeai/app/api/routers/utilities.py:170 draws a fresh seed that is never returned, logged, or accepted back; ExpandPromptRequest/ExpandPromptResponse have no seed field. The route already accepts max_tokens and system_prompt from the client, so accepting an optional seed is zero-risk. Decide explicitly.
• MPS path unverified (no Apple Silicon available). Static review of the .cpu() / .to(device) rewrite is clean and num_samples/replacement survive, but no test covers the branch and the per-token .cpu() hop forces a device sync. Someone on an M-series machine should run one expansion twice with the same seed before merge.

[JPPhoto]

@Pfannkuchensack I believe all of your review points are addressed; please check!

[Pfannkuchensack]

Re-reviewed at 1075233ba0. Five of the six points are genuinely fixed - I verified them by mutation rather than by reading, and the mutations that used to survive now die. Details below, then three new items.

Status of the previous six points

#	Point	Status	How I checked
1	Tests do not prove the feature	Fixed	Both equivalent mutations now killed (see table below)
2	+19.4% runtime per invocation	Fixed	Measured: +2.5 us/token instead of +3.85 ms/token
3	Saved workflows silently become deterministic	Test added, behaviour unchanged and unsignalled	nodeUpdate.test.ts passes; docs still silent
4	func is torch.multinomial misses the method form	Fixed	Mutation killed
5	Expand Prompt cannot reproduce anything	Fixed	Two mutations killed
6	MPS path unverified	Partially	Test exists but covers a non-production path; still skipped here

Mutation results, pytest tests/backend/text_llm tests/app/invocations/test_text_llm_with_preset.py (baseline 33 passed, 1 skipped):

Mutation	Result
generator cache removed (RNG never advances)	killed - 2 failed
LogitsProcessor never arms the seed	killed - 1 failed
torch.Tensor.multinomial left unpatched	killed - 1 failed
Expand Prompt ignores the client seed	killed - 1 failed
Expand Prompt always uses a fixed seed	killed - 1 failed
plain Text LLM node drops the user's seed	SURVIVES - 33 passed
Text LLM (with Preset) node drops the user's seed	killed - 1 failed

On point 2, the rewrite is decisive. Measured on this machine, torch.multinomial over 8 elements, median of 5 x 200k calls:

original torch.multinomial       : 14621 ns
patched, no mode armed           : 15622 ns  (+1.0 us - tax on every other caller)
processor + patched, mode armed  : 17097 ns  (+2.5 us per generated token)

That is ~1550x cheaper than the old TorchFunctionMode design: +0.74 ms at max_tokens=300 and +5.1 ms at the 2048 cap, against ~1.16 s and ~6 s before. Worth putting in the PR description as the resolution.

New findings

1. Medium - the primary Text LLM node's seed wiring is untested

invokeai/app/invocations/text_llm.py, TextLLMInvocation.invoke

Changing seed=self.seed to seed=0 there leaves the entire suite green. The same mutation in TextLLMWithPresetInvocation is caught, because tests/app/invocations/test_text_llm_with_preset.py:74 asserts kwargs["seed"] == 123. So the secondary node is covered and the headline node - the one the PR is named for - is not. A refactor that drops the field, or a merge that resolves a conflict the wrong way, ships a seed input that does nothing, with a fully green suite.

To expose this issue, add a test that invokes TextLLMInvocation with a non-default seed and asserts the value reaches TextLLMPipeline.run, mirroring test_preset_node_loads_content_from_db_and_passes_to_llm.

2. Low - the global torch.multinomial patch is not idempotent, and the one-shot del is unguarded

invokeai/backend/text_llm_pipeline.py (module-level patch, and the finally in _sample_with_seed)

_original_torch_multinomial = torch.multinomial runs at module import with no guard, so re-executing the module top level captures the already-installed wrapper as the "original" and stacks a second layer. Both layers then read the same armed next_mode and both run del _seeded_multinomial_state.next_mode in their finally. Demonstrated:

before reload: torch.multinomial is this module's wrapper: True
after reload : _original_torch_multinomial is the OLD WRAPPER (stacked): True
sampling after reload: CRASH -> AttributeError '_thread._local' object has no attribute 'next_mode'

Two one-line fixes, both worth taking: guard the patch (if torch.multinomial is not _seeded_torch_multinomial:), and make the consume tolerant (_seeded_multinomial_state.next_mode = None, or pop). The tolerant consume also fixes a second edge of the same finally: on the pass-through branch (kwargs.get("generator") is not None) the armed one-shot is deleted without being used, so that token would silently sample unseeded. Not reachable through transformers today - it calls torch.multinomial(probs, num_samples=1) with no generator - but it is the same unguarded consume.

I could not confirm whether dev_reload (jurigged) re-executes module top level the way importlib.reload does, so treat the reload trigger as demonstrated-but-not-necessarily-reachable; the fix is cheap either way.

3. Low - four of the six new RNG tests exercise a code path production never takes

invokeai/backend/text_llm_pipeline.py, _SeededMultinomialMode.__enter__ / __exit__

Production arms the mode exclusively through _SeededMultinomialProcessor setting next_mode. Nothing outside the tests uses with _SeededMultinomialMode(...), so the active slot and both context-manager methods are dead in the shipped path. The tests that pin global-RNG independence, tensor-method coverage, MPS behaviour and - most relevant - concurrent isolation between invocations all go through active, not through next_mode.

That matters because "invocation-local generators prevent concurrent requests from interfering" is a PR summary claim, and the test demonstrating it does not use the mechanism that ships. I checked the real path by hand rather than assume: two concurrent pipeline.run(...) calls against the tiny model with seeds 42 and 1234, five repetitions, each result matches its isolated sequential run. The production path is correctly isolated - this is a coverage gap, not a bug.

To expose this issue, add a test that runs two pipeline.run(...) calls concurrently with different seeds against the tiny model and asserts each matches its isolated single-run output.

[JPPhoto]

@Pfannkuchensack Latest fixes:

• Made global multinomial patching safe across module reloads.
• Prevented stale one-shot RNG state and ensured armed seeds override explicit generators.
• Added coverage for primary Text LLM seed forwarding.
• Added production concurrent-run isolation coverage.
• Added production MPS coverage with constrained-memory cleanup/skip handling.
• Documented that migrated 1.0.0 workflows default seed to 0.