Skip to content

chore(tests): isolate deacon's shared state cache per test process#359

Merged
pofallon merged 1 commit into
mainfrom
chore/isolate-state-cache-in-tests
Jul 24, 2026
Merged

chore(tests): isolate deacon's shared state cache per test process#359
pofallon merged 1 commit into
mainfrom
chore/isolate-state-cache-in-tests

Conversation

@pofallon

Copy link
Copy Markdown
Contributor

Summary

Kills a shared-path flaky-test class found in a sweep (finding #1). deacon's workspace-state cache lives at std::env::temp_dir()/deacon-state/ — a path shared by every process.

The race

DiskCache loads index.json into memory once at open and rewrites the whole index on every set (crates/core/src/state.rscrates/core/src/cache/disk.rs). So two concurrent deacon invocations (e.g. parallel up/down docker tests) each hold a stale in-memory index and clobber the other's entry — last-writer-wins. A later down in a fresh process reloads an index missing its own entry and "loses" the state its up saved. Atomic writes prevent corruption but not this lost update.

smoke_compose_edges.rs already worked around it by hand with a per-test TMPDIR; every other up/down test stayed exposed, and new ones would forget.

Fix (test-side, correctly scoped)

Add support::deacon_command() — a factory that builds the deacon CLI command with TMPDIR/TMP/TEMP redirected to a process-global OnceLock<TempDir>. Under nextest each test is its own process, so the home is unique per test, and a test's up/down/exec share it (as they must). Retrofit the 11 up/down/exec test files that spawn the binary to use it, and drop the hand-rolled workaround.

A product-side fix would mean reworking the shared DiskCache's concurrency model (broad — it also backs the features cache — and risky: a naive index merge would break down's removals), so isolation is the proportionate fix.

Verification

  • Empirical isolation check: with the helper, deacon up wrote 0 files to shared /tmp/deacon-state (stayed at 213) and put its state in the isolated dir instead.
  • make test-nextest-fast — 3123 passed ✓
  • cargo clippy --all-targets --all-features -- -D warnings + cargo fmt --all -- --check
  • Docker spot-checks: integration_down (9/9), smoke_up_idempotent (3/3, real up/down cycles) pass.

Sweep notes (for the curious)

The same sweep cleared the other axes: ports (all binds use ephemeral :0), env races (fully hardened via temp_env), and docker names (already unique via unique_name()/tempdir labels — a couple of latent-but-unreachable fixed image tags noted for optional defense-in-depth). The perf-ceiling flake is addressed separately in #358.

🤖 Generated with Claude Code

deacon's workspace-state cache lives at `std::env::temp_dir()/deacon-state/`
— a path shared across every process. Its on-disk `index.json` is loaded into
memory once and rewritten wholesale on every `set`, so two concurrent `deacon`
invocations (e.g. parallel `up`/`down` docker tests) can clobber each other's
index entry (last-writer-wins). A later `down` in a fresh process then reloads
an index missing its own entry and "loses" the state its `up` saved. One test
(smoke_compose_edges) already worked around this by hand with a per-test
`TMPDIR`; every other up/down test remained exposed.

Add `support::deacon_command()`: a factory that builds the `deacon` CLI command
with `TMPDIR`/`TMP`/`TEMP` redirected to a process-global `OnceLock<TempDir>`.
Under nextest each test is its own process, so the home is unique per test, and
a test's `up`/`down`/`exec` share it (as they must). Retrofit the 11 up/down/
exec test files that spawn the binary to use it, and drop the hand-rolled
workaround. A fix product-side would mean reworking the shared DiskCache's
concurrency (broad, touches the features cache too), so isolation is the
correctly-scoped fix.

Verified: with the helper, `deacon up` writes zero files to the shared
`/tmp/deacon-state` and puts state in the isolated dir instead; fast suite
(3123) green; `integration_down` + `smoke_up_idempotent` docker tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019vhr7Tcf8ybvSZSE1owqBT
@github-actions github-actions Bot added the chore Chore & maintenance label Jul 24, 2026
@pofallon
pofallon merged commit 49fca9f into main Jul 24, 2026
14 checks passed
@pofallon
pofallon deleted the chore/isolate-state-cache-in-tests branch July 24, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Chore & maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant