Skip to content

Preservation doctrine for mixed drafts (behavior rows, not a scanner pattern) - #4

Open
AMPMIO wants to merge 1 commit into
theclaymethod:mainfrom
AMPMIO:preserve-voice-on-mixed-drafts
Open

Preservation doctrine for mixed drafts (behavior rows, not a scanner pattern)#4
AMPMIO wants to merge 1 commit into
theclaymethod:mainfrom
AMPMIO:preserve-voice-on-mixed-drafts

Conversation

@AMPMIO

@AMPMIO AMPMIO commented Jul 26, 2026

Copy link
Copy Markdown

Note: this is the agent-behavior contribution path from CLAUDE.md ("Agent behavior change: add a skill row only when the behavior is rewrite, preserve, decline, or route"), not the scanner FN/FP path the PR template is shaped around. The behavior here is preserve. I've kept the template's sections where they map and said plainly where they don't.

Specimen & source

The specimen is the repo's own eval rationale. Two skill rows already name this gap in their failure_mode fields:

  • SKILL-DONOHARM-01: "Skill has no defined exit for clean text; default mode always rewrites, risking degradation of good writing and invented edits."
  • SKILL-OVEREDIT-01: "B5: 'when in doubt cut' collapses the nuanced it-depends argument into a slogan; only diff_check warns and it's advisory."

Both are judged. Neither is backed by a rule the agent can follow.

The gap

The suite covers the two ends: clean input (do nothing) and slop-dense input (rewrite freely). The common case sits between them — a human paragraph carrying two or three tells.

Every gate in Validation scores whether the output is sloppy. None scores whether the draft still belongs to whoever wrote it. So a wholesale rewrite of a nearly-clean draft passes all of them, and diff_check.py — the only signal that would catch it — is run but is absent from the blocking list.

Coverage

Behavior rows rather than FN/FP/REC:

row id kind split / domain what it pins
SKILL-MIXED-01 behavior (preserve) tune / narrative One throat-clearing opener on an otherwise clean paragraph. Fix the opener, leave the rest.
SKILL-MIXED-02 behavior (preserve) holdout / report Five hard hits wrapped around two blunt first-person admissions and two figures. Rewrite heavily, keep the human signal.

Both carry deterministic backstops alongside the judge assertions, so they aren't judge-only:

  • MIXED-01_difflib_ratio floor of 0.70 plus _banned_phrase_clean. Calibrated before wiring in, per the convention in the SCRIPT_ASSERTIONS comment: deleting the four-word opener scores 0.973; a legitimately tighter edit that keeps the writer's cadence scores 0.753; a neutral corporate rewrite scores 0.406. The floor clears both good edits and fails the rewrite with margin.
  • MIXED-02 — similarity is the wrong instrument when the correct rewrite is heavy, so the figures (4,200, 310) and the blunt admission are asserted directly, plus _banned_phrase_clean.

Fixture behavior, verified before writing the rows: mixed01_original.txt scans 1 hard / 0 soft, and removing just those four words leaves it 0 hard / 0 soft — so the correct edit is provably a deletion. mixed02_original.txt scans 5 hard / 1 soft.

The doctrine

  • SKILL.md — a Preservation section: scope the edit to what the scanners found, note three to five voice traits before Pass 2, treat blunt first-person lines as signal, and return clean text unchanged rather than manufacturing findings.
  • references/commands/rewrite.md — Pass 1 gains the voice-signal step and a clean-input early exit; Pass 2 cites Preservation.
  • SKILL.md Validation — excessive_change promoted from advisory to justify-or-revise when the input scanned mostly clean, which keeps heavy rewrites of genuinely sloppy input unaffected.

This is deliberately scoped so teach/mimic keep owning the stored-profile case. Preservation is a per-draft reading for the far more common no-profile path.

Gate results

run_adversarial.py                  PASS 480  XFAIL 1 (known bugs)  XPASS 0  FAIL 0
build_shared_benchmark.py --check   shared-benchmark.json is up to date.
check_taboo_parity.py               taboo parity ok: 313 banned phrase keys documented
check_pattern_coverage.py           coverage: 83/83 structural patterns, 313/313 banned phrases exercised.
                                    protections: 57/57 categories claimed by a protects FP row.
check_evals_schema.py               evals schema ok: 517 rows, 481 script, 36 skill
check_skill_examples.py             skill examples pass scanner and readability gates
skill-benchmark validate --strict-leakage   OK: unslop — 36 cases, 3 ablations

Identical to the baseline on 749754d (480/1/0), which is expected: run_adversarial.py executes the 481 script rows, and the two new rows are skill targets.

SPAN-03 failed once mid-session and passed on re-run, on this branch and on a stashed HEAD alike. It's a wall-clock cost assertion and nothing here touches scanner cost, so I've treated it as load flake rather than a regression — flagging it in case it's worth a tolerance.

Prose passes the scanners it ships

Both edited files are clean under banned_phrase_scan.py and structure_scan.py --genre docs. Two of my own drafts didn't start that way, which felt like the right kind of dogfooding:

  • The first SKILL.md paragraph tripped anti_slop_register with "Edit it as a mixed draft, not as machine output."
  • The first rewrite.md edit pushed sentence_burstiness to 0.503 (clean at HEAD).

Both rewritten and re-verified.

Confirmation

  • Specimen is the repo's own eval rationale; nothing third-party to redact.
  • Not applicable / not done. The template's red-first proof assumes a scanner row whose output can be pasted. These are judge-scored behavior rows, and run_behavioral.sh tune needs a live model this environment doesn't have — so I have not verified that a model fails these rows before the doctrine change and passes after. That's the one claim in this PR I can't back with output, and it's the thing worth checking before merge. The deterministic backstops were calibrated against hand-written good and bad outputs (numbers above); the judge assertions were not executed.
  • No case marked xfail; FP-06 remains the only documented XFAIL.

Unrelated, happy to file separately

On Python 3.14.5, 250 of 481 rows fall back from in-process to subprocess dispatch with AttributeError: '_io.StringIO' object has no attribute 'buffer'. The suite still passes and the fallback is doing its job, but the in-process runner is covering about a third of what it looks like it covers. Say the word and I'll open an issue with the trace.

Two skill rows already name the gap in their own failure_mode fields:
SKILL-DONOHARM-01 says "the skill has no defined exit for clean text;
default mode always rewrites", and SKILL-OVEREDIT-01 says "only
diff_check warns and it's advisory". Both test the behavior with a judge.
Neither is backed by a rule the agent can follow.

The suite covers clean input (do nothing) and slop-dense input (rewrite
freely). The common case sits between them: a human paragraph carrying
two or three tells. Every gate scores whether the output is sloppy, so a
wholesale rewrite of a nearly-clean draft passes all of them.

Eval-first, per CLAUDE.md:

- SKILL-MIXED-01 (tune, narrative): one throat-clearing opener on an
  otherwise clean paragraph. Deleting the opener scores 0.97 similarity;
  a legitimately tighter edit that keeps the cadence scores 0.75; a
  neutral corporate rewrite scores 0.41. The 0.70 floor clears both good
  edits and fails the rewrite.
- SKILL-MIXED-02 (holdout, report): five hard hits wrapped around two
  blunt first-person admissions and two figures. The rewrite should be
  heavy, so similarity is the wrong instrument; the figures and the
  admission are asserted directly.

Then the doctrine: a Preservation section in SKILL.md, a voice-signal
step and a clean-input early exit in the rewrite flow, and
diff_check's excessive_change promoted from advisory to justify-or-revise
when the input scanned mostly clean.

Checks:

  run_adversarial.py         PASS 480  XFAIL 1  FAIL 0
  build_shared_benchmark.py --check   up to date
  check_taboo_parity.py      313 keys documented
  check_pattern_coverage.py  83/83 structural, 313/313 phrases, 57/57 protections
  check_evals_schema.py      517 rows, 481 script, 36 skill
  skill-benchmark validate --strict-leakage   OK: 36 cases, 3 ablations

Contributed prose passes the scanners it ships: banned_phrase_scan clean
and structure_scan --genre docs clean on both edited files. The first
draft of the SKILL.md paragraph tripped anti_slop_register with a "not X"
contrast and the rewrite.md edit tripped sentence_burstiness; both are
fixed.

run_behavioral.sh tune was not run: the judge rows need a live model this
environment does not have.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant