Skip to content

[Refactor] Refactor tests module and add enhanced citest pipeline #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wangxingjun778 merged 2 commits into from
Aug 6, 2026
Merged

[Refactor] Refactor tests module and add enhanced citest pipeline #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2dd3c57
fix(engine): read the recording step count from the active trajectory
wangxingjun778 Aug 6, 2026
01c6444
test: add a real journey layer with cassette-backed CI tiers
wangxingjun778 Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 82 additions & 7 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,70 @@ concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
# Both lanes below are offline: the LLM boundary is served from the committed
# cassette store. That is deliberate — a pull request from a fork cannot access
# secrets, so anything gating a merge must run without them.
LEAPFLOW_TEST_LLM_MODE: replay

jobs:
test:
# ── L1: pull-request lane ──────────────────────────────────────────────
# Static checks, the whole real layer, and the whole mock layer.
#
# Change-scoped selection is deliberately NOT used here, and the reason is
# arithmetic rather than distrust. Measured on this suite: the full mock layer
# is ~18s, while the always-on tier alone (real journeys + regression ledger +
# architecture contracts) is ~14s. Those tiers can never be selected away, so
# they set the floor — selecting the mock layer can save at most ~4s, however
# precise the selection gets. Spending correctness risk on four seconds is a
# bad trade.
#
# Revisit when the full mock layer stops fitting the feedback budget —
# concretely, when `make test-unit` exceeds ~3 minutes on CI hardware. The
# machinery is ready and tested (tools/impact.py, with a coverage-derived map
# in tests/.impact/); `make test-impact` already uses it locally, where a
# single-module change narrows to 2-3 test files.
#
# The live lane *does* select, in .github/workflows/nightly-live.yaml: there a
# journey costs real tokens, so the arithmetic comes out the other way.
pr:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-extras

- name: Lint
run: uv run ruff check src/leapflow/ tests/ tools/

- name: Verify derived fixtures match the cassette store
run: uv run python tools/sync_fixtures.py --check

- name: Real layer — journeys and always-on guards
run: uv run pytest tests/journeys tests/regression tests/test_architecture_contracts.py -q -m "e2e or invariant or unit" -n 4

- name: Mock layer (full)
run: uv run pytest tests/ -q -m "not e2e" --tb=short -n auto

# ── L2: main lane ──────────────────────────────────────────────────────
# Everything, unscoped, across the supported matrix.
main:
if: github.event_name == 'push'
runs-on: ${{ matrix.os }}
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
Expand All @@ -29,15 +90,29 @@ jobs:

- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-extras

- name: Lint
run: uv run ruff check src/leapflow/ tests/
run: uv run ruff check src/leapflow/ tests/ tools/

- name: Verify derived fixtures match the stored exchanges
run: uv run python tools/sync_fixtures.py --check

- name: Mock layer — full
run: uv run pytest tests/ -q -m "not e2e" --tb=short -n auto

- name: Real layer — full
run: uv run pytest tests/journeys -q -m e2e --tb=short -n 4

- name: Run tests
run: uv run pytest tests/ -q --tb=short
env:
LEAPFLOW_MOCK_HOST: '1'
LEAPFLOW_LLM_API_KEY: 'test-key-ci'
- name: Daemon logs on failure
if: failure()
run: |
echo "Journey daemons log under the scratch root; surface anything left behind."
find /tmp -maxdepth 6 -name 'leapd.log' -newermt '-40 minutes' 2>/dev/null | while read -r log; do
echo "===== $log ====="
tail -n 120 "$log"
done
213 changes: 213 additions & 0 deletions .github/workflows/nightly-live.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
name: Nightly live

# The only lane that talks to a real provider. It exists to catch what replay
# structurally cannot: a provider changing its behavior or its payload shape.
# Everything else runs offline, so a red build here never blocks a merge — it
# tells us the recorded truth has drifted from the real one.

on:
schedule:
# 02:30 UTC daily.
- cron: '30 2 * * *'
workflow_dispatch:
inputs:
rerecord:
description: 'Capture fresh provider traffic and open a PR with it'
type: boolean
default: false
# Opt-in per pull request via the `ci:live` label. Unlike the schedule, this
# path has a diff, so it runs only the journeys the change could plausibly
# break — each live journey costs real tokens and real minutes.
pull_request:
types: [labeled, synchronize, reopened]

concurrency:
group: nightly-live-${{ github.event.pull_request.number || 'schedule' }}
cancel-in-progress: false

permissions:
contents: read

jobs:
# ── L3: real provider ──────────────────────────────────────────────────
live:
# On a pull request, only with the `ci:live` label — never automatically, so a
# fork PR cannot spend tokens.
if: >-
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'ci:live')
runs-on: ubuntu-latest
timeout-minutes: 40
# Credentials live as secrets on this environment, so only jobs that declare
# it can read them. Deliberately *without* required-reviewer or
# deployment-branch rules: reviewers would leave the nightly cron waiting for
# a human, and restricting branches to main would reject every `ci:live` run
# (a pull_request ref is refs/pull/N/merge). The real gates are that fork PRs
# never receive secrets, that applying the label needs write access, and that
# each journey caps its own calls and tokens.
environment: live-llm
steps:
- uses: actions/checkout@v4
with:
# Journey selection needs history to find the merge base.
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-extras

- name: Decide which journeys to run
id: pick
# A scheduled run has no diff and takes every live-capable journey. A
# labelled pull request takes only the journeys whose declared
# SUBJECT_PATHS the change touches. Journeys with LIVE_SIGNAL = False
# (control plane, lifecycle) are excluded either way, and R4 additionally
# refuses to run live because it asserts on injected failures.
env:
BASE_REF: ${{ github.base_ref }}
run: |
if [ -n "${BASE_REF}" ]; then
JOURNEYS=$(uv run python tools/impact.py --base "origin/${BASE_REF}" --live-journeys)
else
JOURNEYS=$(uv run python tools/impact.py --live-journeys)
fi
printf 'selected journeys:\n%s\n' "${JOURNEYS}"
echo "journeys=$(echo ${JOURNEYS} | tr '\n' ' ')" >> "$GITHUB_OUTPUT"

- name: Journeys against the real provider
if: steps.pick.outputs.journeys != ''
env:
LEAPFLOW_TEST_LLM_MODE: live
LEAPFLOW_LLM_API_KEY: ${{ secrets.LEAPFLOW_LLM_API_KEY }}
LEAPFLOW_LLM_BASE_URL: ${{ secrets.LEAPFLOW_LLM_BASE_URL }}
# A cheap model keeps the lane affordable; the journeys assert
# invariants, not prose quality. Each journey also enforces its own
# provider-call *and* token ceilings, so neither a non-converging turn
# nor prompt growth can run up a bill.
LEAPFLOW_LLM_MODEL: ${{ secrets.LEAPFLOW_LLM_CHEAP_MODEL }}
JOURNEYS: ${{ steps.pick.outputs.journeys }}
run: uv run pytest ${JOURNEYS} -q -m e2e --tb=short

- name: Daemon logs on failure
if: failure()
run: |
find /tmp -maxdepth 6 -name 'leapd.log' -newermt '-40 minutes' 2>/dev/null | while read -r log; do
echo "===== $log ====="
tail -n 200 "$log"
done

# ── Re-record: refresh recorded truth and propose it as a diff ──────────
# Manual only. Recorded traffic is a reviewed artefact: a bot silently updating
# what the mock layer asserts against would defeat the point of recording it.
# Recording writes to recordings/ and never touches the replay store, so this
# job cannot break the offline lanes.
rerecord:
if: github.event_name == 'workflow_dispatch' && inputs.rerecord == true
runs-on: ubuntu-latest
timeout-minutes: 40
environment: live-llm
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-extras

- name: Capture real provider traffic
env:
LEAPFLOW_TEST_LLM_MODE: record
LEAPFLOW_LLM_API_KEY: ${{ secrets.LEAPFLOW_LLM_API_KEY }}
LEAPFLOW_LLM_BASE_URL: ${{ secrets.LEAPFLOW_LLM_BASE_URL }}
LEAPFLOW_LLM_MODEL: ${{ secrets.LEAPFLOW_LLM_CHEAP_MODEL }}
run: uv run pytest tests/journeys -q -m e2e --tb=short

- name: Derive mock-layer response shapes from the new traffic
run: uv run python tools/sync_fixtures.py

- name: Confirm the offline lanes still pass
env:
LEAPFLOW_TEST_LLM_MODE: replay
run: uv run pytest tests/journeys tests/regression -q -m "e2e or invariant" -n 4

- name: Open a pull request with the refreshed traffic
uses: peter-evans/create-pull-request@v6
with:
branch: chore/rerecord-provider-traffic
title: 'chore(tests): refresh recorded provider traffic'
body: |
Captured fresh provider traffic and re-derived the response shapes the
mock layer checks against.

Review the diff in `tests/_fixtures/llm_responses/response_shapes.json`
first: a change there means a provider altered its payload shape, and
some parser may now be reading a field that no longer exists.
commit-message: 'chore(tests): refresh recorded provider traffic'
add-paths: |
tests/_fixtures/recordings/**
tests/_fixtures/llm_responses/**

# ── Refresh the impact map from a full green run ────────────────────────
impact-map:
# Never on a pull request: the map is a repository artefact refreshed from a
# full green run, not something a PR should regenerate.
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install uv
uses: astral-sh/setup-uv@v4
with:
enable-cache: true

- name: Install dependencies
run: uv sync --all-extras

- name: Rebuild the coverage-derived impact map
env:
LEAPFLOW_TEST_LLM_MODE: replay
run: uv run python tools/impact.py --build-map

- name: Open a pull request with the refreshed map
uses: peter-evans/create-pull-request@v6
with:
branch: chore/refresh-impact-map
title: 'chore(tests): refresh the coverage-derived impact map'
body: |
Regenerated `tests/.impact/coverage_map.json` from a full green run.

This map is what lets the pull-request lane scope the mock layer to
the change while still seeing runtime coupling through EventBus and
Protocol indirection.
commit-message: 'chore(tests): refresh the coverage-derived impact map'
add-paths: tests/.impact/coverage_map.json
23 changes: 22 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,33 @@ This document is the LeapFlow engineering collaboration contract. It is not only

## Testing Philosophy

The suite has **two layers with different jobs**, and keeping the boundary sharp is what stops each from doing the other's work badly.

**Mock layer** (`tests/*.py`, marked `unit`/`component`) — broad and fast. It owns pure algorithms, state-machine branches, error-classification tables, rare and malformed inputs, and single-module invariants. Branch combinations can only be enumerated here, and only here is the feedback measured in milliseconds.

**Real layer** (`tests/journeys/`, marked `e2e`) — six coarse journeys driving a real `leapd` subprocess over RPC, with the LLM boundary served by a local cassette proxy. It owns what a mock structurally cannot observe: cross-module wiring, process boundaries, session identity, workspace binding, real persistence, and pushed runtime metadata. Every incident recorded in this document shipped with a green mock suite.

Three rules keep the split honest:

1. **Anything provable with one mock-layer assertion must not enter the real layer.**
2. **One journey is one test case.** Express variation as ordered phases inside a single session; never parameterize a journey.
3. **The real layer has a hard case budget** (`tests/regression/test_suite_budget.py`). When it is reached, merge a journey — do not raise the ceiling. The budget is what lets the real layer run on *every* push, and a suite that can be skipped will be skipped.

The two layers are joined by recorded traffic. `tests/_fixtures/cassettes/` holds the deterministic inputs the offline lanes replay (rebuilt with `make seed-cassettes`); `tests/_fixtures/recordings/` holds real provider traffic captured by `make record-traffic`. `make sync-fixtures` distils both into `tests/_fixtures/llm_responses/response_shapes.json`, which the mock layer asserts against — so a provider dropping a field turns the build red instead of passing forever against a body written from memory. Recording never writes to the replay store: a multi-turn agent conversation cannot be replayed from a recording, because each turn's prompt embeds the exact round-by-round history of the turns before it.

Each journey also declares two cost ceilings, both enforced at the proxy and reported by `finish()`. `max_llm_calls` is the convergence guard: a turn that stops converging is cut off instead of running to the engine's iteration cap. `max_llm_tokens` is the cost guard, and it catches what call count cannot — prompt growth raises the bill without adding a single round. Raising a ceiling is not the fix for hitting one.

**Which tests run.** The offline lanes never select: the mock layer runs in full, and the real journeys run in full on every push. Selection would save at most a few seconds, because the always-on tiers set the floor, and a suite that can be skipped will be skipped. The *live* lane is the exception — there a journey costs real tokens, so it selects: each journey declares `SUBJECT_PATHS` (the sources it exercises) and `LIVE_SIGNAL` (whether a real provider adds signal), and `tools/impact.py --live-journeys` picks from those. Declaring them inside the journey keeps that knowledge next to the assertions it describes. `tools/impact.py` can also scope the mock layer (`make test-impact`) for local work on a large change; it is not wired into CI.

- **Unit tests must be hermetic**: no network, no LLM calls
- **Journeys must not mock anything**: a journey that reaches for `unittest.mock` has become an expensive unit test
- **Provider bodies are recorded, not written**: use a cassette or a derived fixture; a hand-authored body keeps passing after the provider changes
- **Faked construction needs real-instance cover**: `object.__new__` plus private-attribute assignment is acceptable only for ordering contracts, and only when the same file also builds the class properly and drives the production path
- **py_compile all modified files**: syntax errors caught before test run
- **Import chain verification**: every new module must be importable standalone
- **Existing tests must not regress**: all tests must pass after every change
- **User-facing flows must not regress**: preserve or improve usability, feedback clarity, and failure recovery for impacted paths
- **Verification sequence**: compile → import → unit test → integration (if applicable)
- **Verification sequence**: compile → import → mock layer → real journeys
- **Behavior contracts over snapshots**: assert invariants, not frozen values
- **Mock at boundaries only**: mock external I/O (network, disk), never internal logic
- **A test may not fabricate the wiring it claims to cover**: building an object with `object.__new__` and assigning the private attributes the code reads cannot detect a wrong attribute *name* — the test simply agrees with the typo. Calibration tests did exactly that and stayed green while every real turn raised `AttributeError`. Any test whose stated purpose is wiring must construct the real object and drive the production path.
Expand Down
Loading
Loading