[Refactor] Refactor tests module and add enhanced citest pipeline - #29
Merged
Conversation
`SessionController.recording_step_count` read `recorder.step_count`, which does not exist: the count lives on the trajectory the recorder is filling. Every `/teach status` issued while a recording was active raised AttributeError instead of reporting progress. Reads through `recorder.current_trajectory.step_count` and coerces the result, so the property returns 0 both before a recording starts and if the recorder has yet to open a trajectory.
Every incident in AGENTS.md shipped with a green mock suite: 1531 hermetic cases cannot observe cross-module wiring, process boundaries, session identity, or pushed runtime metadata, because they never cross them. This adds the layer that can, without giving up the fast one. Six coarse journeys (tests/journeys/) drive a real leapd subprocess over RPC with the LLM boundary served by a local OpenAI-compatible proxy (tests/_harness/cassette_proxy.py). A proxy rather than a patched provider: leapd runs as a separate process, so in-process patching cannot reach it, and addressing the boundary through LEAPFLOW_LLM_BASE_URL keeps the real SDK, httpx stack, SSE framing and retry classification in the path. Four modes select the lane: replay (offline default), seed (build the replay store from a journey's script, no credential needed), record (capture real traffic as wire-shape evidence), live (real provider, persists nothing). Recording writes to a separate store on purpose — a multi-turn conversation cannot be replayed from a recording, since turn n's prompt embeds the round-by-round history of turns 1..n-1, so one divergence cascades. CI gains three tiers. The PR and main lanes stay fully offline, which is a requirement rather than a convenience: a fork pull request cannot read secrets, so anything gating a merge has to run without them. Only nightly-live.yaml reaches a provider, and it selects journeys via tools/impact.py because there each one costs real tokens. Cost is bounded by construction, not by trust: each journey declares max_llm_calls (convergence) and max_llm_tokens (prompt growth, which call count cannot see), both enforced at the proxy and returned as non-retryable 400s so a runaway loop stops at the ceiling instead of feeding the provider's retry logic. tests/regression/ holds the always-on guards — a hard journey budget (merge, never raise), an incident ledger, provider-shape drift detection, and meta-tests against suite degradation. Also updates AGENTS.md with the two-layer contract and the rules that keep the split honest, and tests/README.md with the credential setup: three environment secrets under live-llm, and why both protection rules must stay off (required reviewers would strand the cron; a main-only branch rule rejects refs/pull/N/merge).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.