Preservation doctrine for mixed drafts (behavior rows, not a scanner pattern) - #4
Open
AMPMIO wants to merge 1 commit into
Open
Preservation doctrine for mixed drafts (behavior rows, not a scanner pattern)#4AMPMIO wants to merge 1 commit into
AMPMIO wants to merge 1 commit into
Conversation
Two skill rows already name the gap in their own failure_mode fields: SKILL-DONOHARM-01 says "the skill has no defined exit for clean text; default mode always rewrites", and SKILL-OVEREDIT-01 says "only diff_check warns and it's advisory". Both test the behavior with a judge. Neither is backed by a rule the agent can follow. The suite covers clean input (do nothing) and slop-dense input (rewrite freely). The common case sits between them: a human paragraph carrying two or three tells. Every gate scores whether the output is sloppy, so a wholesale rewrite of a nearly-clean draft passes all of them. Eval-first, per CLAUDE.md: - SKILL-MIXED-01 (tune, narrative): one throat-clearing opener on an otherwise clean paragraph. Deleting the opener scores 0.97 similarity; a legitimately tighter edit that keeps the cadence scores 0.75; a neutral corporate rewrite scores 0.41. The 0.70 floor clears both good edits and fails the rewrite. - SKILL-MIXED-02 (holdout, report): five hard hits wrapped around two blunt first-person admissions and two figures. The rewrite should be heavy, so similarity is the wrong instrument; the figures and the admission are asserted directly. Then the doctrine: a Preservation section in SKILL.md, a voice-signal step and a clean-input early exit in the rewrite flow, and diff_check's excessive_change promoted from advisory to justify-or-revise when the input scanned mostly clean. Checks: run_adversarial.py PASS 480 XFAIL 1 FAIL 0 build_shared_benchmark.py --check up to date check_taboo_parity.py 313 keys documented check_pattern_coverage.py 83/83 structural, 313/313 phrases, 57/57 protections check_evals_schema.py 517 rows, 481 script, 36 skill skill-benchmark validate --strict-leakage OK: 36 cases, 3 ablations Contributed prose passes the scanners it ships: banned_phrase_scan clean and structure_scan --genre docs clean on both edited files. The first draft of the SKILL.md paragraph tripped anti_slop_register with a "not X" contrast and the rewrite.md edit tripped sentence_burstiness; both are fixed. run_behavioral.sh tune was not run: the judge rows need a live model this environment does not have.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Specimen & source
The specimen is the repo's own eval rationale. Two
skillrows already name this gap in theirfailure_modefields:SKILL-DONOHARM-01: "Skill has no defined exit for clean text; default mode always rewrites, risking degradation of good writing and invented edits."SKILL-OVEREDIT-01: "B5: 'when in doubt cut' collapses the nuanced it-depends argument into a slogan; only diff_check warns and it's advisory."Both are judged. Neither is backed by a rule the agent can follow.
The gap
The suite covers the two ends: clean input (do nothing) and slop-dense input (rewrite freely). The common case sits between them — a human paragraph carrying two or three tells.
Every gate in
Validationscores whether the output is sloppy. None scores whether the draft still belongs to whoever wrote it. So a wholesale rewrite of a nearly-clean draft passes all of them, anddiff_check.py— the only signal that would catch it — is run but is absent from the blocking list.Coverage
Behavior rows rather than FN/FP/REC:
SKILL-MIXED-01SKILL-MIXED-02Both carry deterministic backstops alongside the judge assertions, so they aren't judge-only:
_difflib_ratiofloor of 0.70 plus_banned_phrase_clean. Calibrated before wiring in, per the convention in theSCRIPT_ASSERTIONScomment: deleting the four-word opener scores 0.973; a legitimately tighter edit that keeps the writer's cadence scores 0.753; a neutral corporate rewrite scores 0.406. The floor clears both good edits and fails the rewrite with margin.4,200,310) and the blunt admission are asserted directly, plus_banned_phrase_clean.Fixture behavior, verified before writing the rows:
mixed01_original.txtscans 1 hard / 0 soft, and removing just those four words leaves it 0 hard / 0 soft — so the correct edit is provably a deletion.mixed02_original.txtscans 5 hard / 1 soft.The doctrine
SKILL.md— aPreservationsection: scope the edit to what the scanners found, note three to five voice traits before Pass 2, treat blunt first-person lines as signal, and return clean text unchanged rather than manufacturing findings.references/commands/rewrite.md— Pass 1 gains the voice-signal step and a clean-input early exit; Pass 2 citesPreservation.SKILL.mdValidation —excessive_changepromoted from advisory to justify-or-revise when the input scanned mostly clean, which keeps heavy rewrites of genuinely sloppy input unaffected.This is deliberately scoped so
teach/mimickeep owning the stored-profile case.Preservationis a per-draft reading for the far more common no-profile path.Gate results
Identical to the baseline on
749754d(480/1/0), which is expected:run_adversarial.pyexecutes the 481 script rows, and the two new rows areskilltargets.SPAN-03failed once mid-session and passed on re-run, on this branch and on a stashedHEADalike. It's a wall-clock cost assertion and nothing here touches scanner cost, so I've treated it as load flake rather than a regression — flagging it in case it's worth a tolerance.Prose passes the scanners it ships
Both edited files are clean under
banned_phrase_scan.pyandstructure_scan.py --genre docs. Two of my own drafts didn't start that way, which felt like the right kind of dogfooding:SKILL.mdparagraph trippedanti_slop_registerwith "Edit it as a mixed draft, not as machine output."rewrite.mdedit pushedsentence_burstinessto 0.503 (clean atHEAD).Both rewritten and re-verified.
Confirmation
run_behavioral.sh tuneneeds a live model this environment doesn't have — so I have not verified that a model fails these rows before the doctrine change and passes after. That's the one claim in this PR I can't back with output, and it's the thing worth checking before merge. The deterministic backstops were calibrated against hand-written good and bad outputs (numbers above); the judge assertions were not executed.xfail;FP-06remains the only documented XFAIL.Unrelated, happy to file separately
On Python 3.14.5, 250 of 481 rows fall back from in-process to subprocess dispatch with
AttributeError: '_io.StringIO' object has no attribute 'buffer'. The suite still passes and the fallback is doing its job, but the in-process runner is covering about a third of what it looks like it covers. Say the word and I'll open an issue with the trace.