feat(sleep): per-night evidence chain + live prompt registry#151
Merged
Yif-Yang merged 2 commits intoJul 21, 2026
Conversation
This was referenced Jul 19, 2026
* skillopt_sleep/evidence.py — append-only, thread-safe, redacted evidence.jsonl per night: harvest sessions -> miner exchanges (verbatim prompt/reply) -> mined tasks with checks -> split assignment -> every replay attempt (phase-tagged, cache hits marked) -> per-task scores with failing checks named -> reflect exchanges + parsed edits -> gate trials and the final decision with its score arithmetic -> staged artifacts. Config-gated (evidence_log, default on; evidence_max_chars cap). * skillopt_sleep/prompts.py — central registry of the four LLM prompt templates (miner/attempt/judge/reflect), byte-identical defaults to the previously inlined strings; user overrides in prompts.json take effect on the next model call (mtime-checked), no restart needed. * cycle.py builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report; staging.new_staging_dir() de-collides same-second runs; latest_staging() now skips non-adoptable (evidence-only) folders. * tests/test_sleep_evidence.py — 8 no-network stdlib tests: chain completeness, redaction/truncation/ordering, disable flag, prompt override round-trip + live effect, no-tasks-night adoption guard.
Yif-Yang
force-pushed
the
feat/sleep-evidence-chain
branch
from
July 21, 2026 17:26
7829552 to
b12b54c
Compare
There was a problem hiding this comment.
Pull request overview
Adds a per-night, append-only evidentiary chain (evidence.jsonl) to make SkillOpt-Sleep nights fully auditable, and introduces a centralized prompt registry with live, file-based overrides so miner/attempt/judge/reflect templates become versionable and tunable without code changes.
Changes:
- Introduce
EvidenceLogand wire evidence events across harvest → mine → replay → reflect/gate → staging. - Add
skillopt_sleep.promptsprompt registry and refactor backend/miner to render prompts from the registry (with live overrides). - Improve staging lifecycle (pre-created staging dir, de-collide naming, and make
latest_staging()skip non-adoptable evidence-only nights).
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sleep_evidence.py | Adds unit tests for evidence logging and prompt override behavior. |
| tests/test_sleep_engine.py | Updates a mock patch target to the new build_backend entrypoint. |
| skillopt_sleep/staging.py | Adds unique staging-dir helper and prevents adopting evidence-only folders. |
| skillopt_sleep/replay.py | Logs per-task replay results into evidence chain (phase-tagged). |
| skillopt_sleep/prompts.py | New live prompt registry with defaults + override file (mtime-checked). |
| skillopt_sleep/llm_miner.py | Moves miner prompt into registry and logs miner exchanges/tasks into evidence. |
| skillopt_sleep/evidence.py | New thread-safe JSONL evidence logger + reader + backend attachment helpers. |
| skillopt_sleep/cycle.py | Creates evidence log per night, pre-creates staging dir, dual-backend support wiring. |
| skillopt_sleep/consolidate.py | Adds evidence phase tagging and logs gate baseline/trials/decision formula. |
| skillopt_sleep/config.py | Adds config knobs for evidence logging and dual-backend split. |
| skillopt_sleep/backend.py | Uses prompt registry for attempt/judge/reflect and logs cached/model calls + reflect exchanges. |
| docs/sleep/README.md | Documents evidence logging behavior and retention/disable guidance. |
| CHANGELOG.md | Notes new evidence chains and prompt registry in Unreleased additions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+83
to
+90
| def new_staging_dir(project: str) -> str: | ||
| """A staging path that is unique even for two runs in the same second.""" | ||
| base = os.path.join(staging_root(project), _ts_dir()) | ||
| out, i = base, 2 | ||
| while os.path.exists(out): | ||
| out = f"{base}-{i}" | ||
| i += 1 | ||
| return out |
Comment on lines
+166
to
+170
| ev = EvidenceLog( | ||
| ev_path, | ||
| max_chars=int(cfg.get("evidence_max_chars", 4000) or 4000), | ||
| redact=bool(cfg.get("redact_secrets", True)), | ||
| ) |
Comment on lines
+161
to
+162
| ev_path = os.path.join( | ||
| cfg.state_dir, "evidence", f"dryrun-{_ts_dir()}.jsonl") |
Comment on lines
+117
to
+123
| def _run(self, **cfg_extra): | ||
| proj = tempfile.mkdtemp() | ||
| home = tempfile.mkdtemp() | ||
| cfg = load_config( | ||
| invoked_project=proj, projects="invoked", backend="mock", | ||
| claude_home=os.path.join(home, ".claude"), auto_adopt=False, | ||
| **cfg_extra) |
Comment on lines
+196
to
+200
| proj = tempfile.mkdtemp() | ||
| home = tempfile.mkdtemp() | ||
| cfg = load_config( | ||
| invoked_project=proj, projects="invoked", backend="mock", | ||
| claude_home=os.path.join(home, ".claude")) |
Comment on lines
+194
to
+196
| os.makedirs(os.path.dirname(path) or ".", exist_ok=True) | ||
| with open(path, "w", encoding="utf-8") as f: | ||
| json.dump(current, f, ensure_ascii=False, indent=2) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-night evidence chain + live prompt registry for SkillOpt-Sleep
Why (origin of this change)
This started from a user-side audit of what a completed sleep night actually lets you verify. Running real nights (DeepSeek via the
openai_compatiblebackend) against real transcripts, the report tells you that the miner processed N conversations and what rules the reflect step added — but the evidential chain in between is invisible:None of the intermediate artifacts (the miner's verbatim exchange, which checks each task carried, which replay attempt failed which check, the reflect model's raw reply before parsing, the gate's arithmetic) were recorded anywhere. Given how strict/fragile the acceptance gate is by design (strict improvement or nothing), a rejected night was essentially unexplainable after the fact.
The motivation goes beyond auditability. Once every miner/reflect exchange is persisted per night, three things become possible that aren't today:
What
skillopt_sleep/evidence.py— append-only, thread-safe, secret-redactedevidence.jsonlwritten into each night's staging dir. The full chain is logged: harvested sessions → miner exchanges (verbatim prompt/reply) → mined tasks with their checks → train/val split assignment → every replay model call (phase-tagged; cache hits marked key-only) → per-task scores with failing checks named → reflect exchanges + parsed edits → gate baseline/trials and the final decision with its score arithmetic spelled out → staged artifacts. Config-gated:evidence_log(default on),evidence_max_charstruncation cap.skillopt_sleep/prompts.py— central registry of the four LLM prompt templates (miner / attempt / judge / reflect). Defaults are byte-identical to the previously inlined strings (covered by a test). User overrides inprompts.jsontake effect on the next model call (mtime-checked) — no restart. This is what makes the prompts first-class, versionable, and eventually gate-optimizable objects.cycle.py— builds dual backends from config (optimizer_*/target_*), pre-creates the staging dir so evidence lands beside the report;staging.new_staging_dir()de-collides same-second runs;latest_staging()skips non-adoptable (evidence-only) folders so a no-tasks night can never be accidentally adopted.Tests
tests/test_sleep_evidence.py— 8 pure-stdlib, deterministic, no-network tests: chain completeness end-to-end on the mock backend, redaction/truncation/ordering, corrupt-line tolerance,evidence_log=Falsedisable path, prompt-default byte-parity, override round-trip + live effect without restart, and the no-tasks-night adoption guard.Full suite: no new failures vs pristine
main(same 11 pre-existing research-suite loader errors on both; this branch adds 8 tests, all passing).Live smoke: a real DeepSeek night through the
openai_compatiblebackend produced a 95-event chain in which the gate's rejection is fully reconstructable from the log, including its formula (baseline = 0.5·0.000 + 0.5·0.333 = 0.167; candidate = 0.167 → no strict improvement → reject).Series
This is part 1 of a 3-PR series: this PR (evidence chain + prompt registry, no UI) → #152 (a stdlib-only local dashboard that renders the chain and edits the prompts) → #153 (a new harvest source, per the "more sources" direction of #97).