Skip to content

fix(authoritative-text-rules,done-check,quality-list,todo-check,implement,finding-to-audit,review-pipeline,review-pipeline-coderabbit): audit text the agent executes as instructions - #146

Merged
ultimatile merged 3 commits into
mainfrom
feat/145-authoritative-text-rule
Jul 28, 2026
Merged

fix(authoritative-text-rules,done-check,quality-list,todo-check,implement,finding-to-audit,review-pipeline,review-pipeline-coderabbit): audit text the agent executes as instructions#146
ultimatile merged 3 commits into
mainfrom
feat/145-authoritative-text-rule

Conversation

@ultimatile

@ultimatile ultimatile commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Closes #145.

What this adds

authoritative-text-rules, a rule-set SSOT for text an agent executes as instructions and that describes no artifact separate from itself — skill bodies, rule-definition files, and repository agent-instruction files. It holds five items, each written as a sweep procedure rather than a quality predicate: case-space-totality, single-reading, clause-composition, executor-fitness, consumer-closure.

The sweep form is the load-bearing choice. A bottom-up pass over the 106 findings #145 identifies placed 14 under referential and parse indeterminacy and 9 under executor mismatch; the remaining 83 are shapes a compiler, a type checker, or a test suite catches in code and nothing catches in prose. For those 83, what is missing is an oracle rather than a rule concept, so the item states the enumeration the auditor performs and that enumeration stands in for the missing oracle. The other two items take the same form for a different reason: a fresh-context auditor is not primed for an ambiguity or a mismatched executor either, so the enumeration is what makes those items executable rather than aspirational.

done-check gains a second, conditionally dispatched fresh-context auditor beside the quality-list one. It fires when any path in the diff could be text an agent executes as instructions, and when unsure. The rule approximates because it has to: membership is a property of a file's content, while the dispatch decision is made from the diff's path list before any file is opened, so the only choice is which way it errs — a missed dispatch is an audit that silently did not happen, a needless one costs one subagent returning not-applicable rows. Auditor 2's returned rows are checked against the slug set its SSOT's index predicts.

Both auditor prompts now take the rule-set root and the target root as two paths rather than one. The single path silently assumed the rules were vendored inside the audited project; the two-path shape lets a marketplace install pass a different rule-set root, though finding out what that root is remains unaddressed — see the open findings. The three verdicts — pass, concern, and not-applicable — have one definition, in the merge step, which both prompts point at.

Plan-vs-actual delta

The plan in #145 said the SSOT would be one file with no items/ split. It is split, and four further changes go beyond it.

  • Split into SKILL.md plus items/<slug>.md. quality-list's no-hardcoded-slug-list rule requires the coverage check to derive its expected set by reading an index, and main context can read an index but not an item body. Reading a single file loads both, so the split is what makes the coverage check possible at all.
  • todo-check, quality-list, implement, finding-to-audit, and both review pipelines are edited. The plan listed only the manifest and the README as collateral. Making done-check's rule set conditional stranded unconditional statements of it across those files.
  • The existing auditor-1 prompt was moved to two roots as well. The plan deferred that. Once the step resolved two roots, leaving one prompt single-rooted was incoherent within the same step.
  • finding-to-audit's item-format template and host list were rewritten, not just its routing row. Removing done-check from the host table left the template's "For done-check:" heading and the step-5 host list without a referent.
  • The coverage-check mismatch reports rather than halts. Its first form ended the audit without a table, which created a terminal state three callers had to branch on and a verdict class the pipelines' three-value loop exit could not close. Reporting the mismatch as a cross-cutting concern removes that exit and keeps the mismatch blocking inside done-check, though not in the callers — see the open findings.

Verification

The pre-commit hook (mdformat with the gfm and frontmatter plugins) reports a pass on both commits. done-check ran against this diff six times, dispatching both auditors each time — the change is itself authoritative text, so the new rule set audited the diff that defines it. Every one of those six iterations returned concerns; all were resolved, and triage classified none of them a false positive. Several were defects the fixes themselves introduced — rule text whose repair opens a new seam with the rule text around it, which is the failure #145 attributes to this repository's diffs. The Claude code-review gate ran three times, returning six findings, then five, then six; every round's findings were resolved except four of the third round's, which are recorded below. Codex reviewed the branch twice and returned no findings on the second pass.

Known open findings

Four of these came out of the code-review gate's third round and the last from the done-check loop; all five are landing unresolved. #147 tracks them together with the deferrals below. The loops stopped converging: the gate's three rounds returned six, five, and six findings, most of each round's traceable to the previous round's fixes, and in the done-check loop two rows that had passed twice came back as concerns against text neither round had changed. Fixing was regenerating the class about as fast as it closed it, so it was stopped rather than continued.

  • <RULES_ROOT> must be resolved before dispatch, and the text says it differs from the working directory in a marketplace install, but gives no way to find it. todo-check has the same gap for its own root, so this is not new.
  • A cross-cutting concern blocks done-check's own step 5 but is explicitly not a row, and both review pipelines gate on rows. A caller can exit clean with one unresolved.
  • The tie-break graph covers six of the ten item pairs. single-reading and clause-composition state none against executor-fitness or consumer-closure, so one ambiguous dispatch step can yield two concerns for a single defect.
  • The quoted-clause requirement on concerns applies only to auditor 2, though the reason for it — the triager cannot open the item body — holds for auditor 1's rows too.
  • executor-fitness's trigger list is not marked illustrative, unlike its siblings'.

Deferred

  • Symmetric wiring into todo-check. The longest run in the local telemetry log, at 44 code-review rounds, was the change that mirrored done-check's lane split into it.
  • done-check's quality-list rows still have no coverage check while todo-check's do. Closing it changes an existing gate's behaviour.
  • Both review pipelines invoke done-check "in delta mode", which done-check defines nowhere.
  • quality-list/items/ported-code-attribution.md refers to "item 11", a numbering scheme the index no longer uses.

#145's success measure — this repository's code-review iterations per run falling toward the sub-1.8 band, and the findings carrying code-review as their own injected_at_gate shrinking — cannot be measured here. It needs future runs.

…ment,finding-to-audit,review-pipeline,review-pipeline-coderabbit): audit text the agent executes as instructions

Add `authoritative-text-rules`, a rule-set SSOT for text an agent
executes as instructions and that describes no separate artifact —
skill bodies, rule-definition files, agent-instruction files. Five
items: case-space-totality, single-reading, clause-composition,
executor-fitness, consumer-closure. Each is written as a sweep
procedure rather than a quality predicate, because in code a
compiler, type checker, or test suite catches these shapes and in
prose nothing does; the enumeration is the substitute oracle.

Wire it into `done-check`: a second, conditionally dispatched
fresh-context auditor alongside the `quality-list` one, firing when
any diff path could be text an agent executes as instructions and
when unsure. The firing rule over-approximates because the
membership predicate is a property of file content while a dispatch
decision has only paths; a missed dispatch is a silently skipped
audit, a needless one costs a subagent returning not-applicable
rows. Auditor 2's returned rows get a coverage check against the
index-predicted slug set.

Both auditor prompts now take the rule-set root and the target root
as separate paths, so a marketplace install auditing another
checkout resolves correctly. The verdict vocabulary has one
definition, in the merge step, which both prompts point at.

Callers whose unconditional statements the change qualifies:
`todo-check` and `implement` now say the preflight covers one rule
set of two; `quality-list` scopes its index and lane tags to its own
items; `finding-to-audit` routes a promoted rule to a rule-set SSOT
rather than to the runner, and drops the item format that no longer
exists; both review pipelines name the actual host.
…th directions

An item body names no runner, so a rule set built only from outbound
references made every item-only diff a single unit and returned
not-applicable — hiding the runner-versus-item conflict the item
exists to catch.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new rule-set SSOT (authoritative-text-rules) to audit “authoritative text” (instructional prose that is itself the executed artifact), and wires it into done-check as a conditionally-dispatched second fresh-context auditor alongside the existing quality-list audit. It also updates supporting workflow skills and manifests to reflect that done-check can now apply multiple rule sets.

Changes:

  • Add authoritative-text-rules SSOT with an index + five new item files for auditing authoritative text.
  • Update done-check to dispatch a second auditor when the diff paths suggest authoritative text may be present, and to merge those rows.
  • Update workflow docs/skills (todo-check, pipelines, finding-to-audit, README) and marketplace manifest to reference the new rule set.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
skills/todo-check/SKILL.md Clarifies todo-check preflights quality-list only (not the new authoritative-text audit).
skills/review-pipeline/SKILL.md Updates /finding-to-audit guidance to target SSOT rule sets or domain audit skills.
skills/review-pipeline-coderabbit/SKILL.md Same /finding-to-audit guidance update as review-pipeline.
skills/quality-list/SKILL.md Clarifies “this rule set only” semantics and lane tagging when runners apply multiple rule sets.
skills/implement/SKILL.md Notes todo-check doesn’t preview non-quality-list rule sets that done-check may apply later.
skills/finding-to-audit/SKILL.md Updates host table and rule-drafting guidance for SSOT-style rule sets (incl. authoritative-text-rules).
skills/done-check/SKILL.md Adds conditional authoritative-text auditor, row merge rules, and coverage checking behavior.
skills/authoritative-text-rules/SKILL.md New SSOT defining scope/ownership boundary and indexing five authoritative-text audit items.
skills/authoritative-text-rules/items/case-space-totality.md New item: totality of case→outcome mappings in authoritative text.
skills/authoritative-text-rules/items/single-reading.md New item: ambiguity / multi-reading sweep for authoritative text changes.
skills/authoritative-text-rules/items/clause-composition.md New item: contradiction/shadowing/drift across authoritative-text units.
skills/authoritative-text-rules/items/executor-fitness.md New item: checks executor inputs suffice for demanded judgments.
skills/authoritative-text-rules/items/consumer-closure.md New item: ensures produced values/obligations are consumed/terminated.
README.md Adds authoritative-text-rules to the skill list and updates done-check description.
.claude-plugin/marketplace.json Bumps version and registers the new authoritative-text-rules skill path.
Comments suppressed due to low confidence (1)

skills/done-check/SKILL.md:163

  • Step 4’s coverage check reads authoritative-text-rules/SKILL.md without anchoring it to <RULES_ROOT>. When <RULES_ROOT> and <TARGET_ROOT> differ (marketplace / symlink install), this path will be resolved against the target project and the coverage check can’t run.
   **Coverage check on auditor 2's rows.** Only when it was dispatched. Read `authoritative-text-rules/SKILL.md`'s Items index for the predicted slug set — the index, never an item body, and never a slug list written into this file. A return that is exactly one row per predicted slug and no others passes. Any other return — a slug missing, duplicated, or outside the set — gets one re-dispatch of auditor 2 with its prompt unchanged; if the second return still does not match, carry the mismatch as a cross-cutting concern naming the offending slugs. It closes through step 5 like any other concern, so an incomplete rule set blocks completion without introducing an exit of its own.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread skills/done-check/SKILL.md Outdated
Comment thread skills/done-check/SKILL.md
…-file read to it

Step 2 resolved the two roots but only its own prompts used them:
Step 3's contextual reads and Step 4's coverage check still named
rule files by bare relative path, which resolves against the audited
project. Under a marketplace or symlinked install those are a
different tree, so both reads fail. Resolution moves to Step 0 and
every rule-file read is anchored.

Auditor 1's prompt also now labels each returned row with the item's
index slug, as auditor 2's already did, so the merge does not rely
on row ordering.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

skills/done-check/SKILL.md:16

  • Step 0 introduces <RULES_ROOT> / <TARGET_ROOT> separation but never defines how to discover <RULES_ROOT>. In a marketplace / symlink install where cwd is <TARGET_ROOT>, the rest of the procedure can’t be executed as written because none of the rule-set file paths can be resolved.
0. **Resolve the active rule sets.** Resolve two absolute paths first, and use them everywhere below: `<RULES_ROOT>`, the repo / package root holding the `skills/quality-list/` and `skills/authoritative-text-rules/` directories, and `<TARGET_ROOT>`, the project under audit (cwd). They coincide only when the rule sets are vendored inside the target; in a marketplace or symlinked install they differ, and every read of a rule file — main context's as much as a subagent's — resolves against `<RULES_ROOT>`.

   `quality-list` always applies: its base items live in `<RULES_ROOT>/skills/quality-list/SKILL.md`, and language-specific addenda at `<RULES_ROOT>/skills/quality-list/lang-<language>.md` realize them concretely. `authoritative-text-rules` applies conditionally, by the firing rule in Step 2.

   Detect language from the project's `CLAUDE.md` `Language:` declaration; otherwise auto-detect from diff file extensions (`.rs` → rust, `.cpp`/`.cc`/`.cxx`/`.h`/`.hpp` → cpp, `.py` → python, `.ts`/`.tsx` → typescript, `.go` → go, etc.). Multi-language projects detect every present language; each matching addendum applies. Missing addendum → base rules only for that language (not a concern). Step 0 only **detects** the language(s); it routes nothing. Each consumer that applies `quality-list` — the Step 2 mechanical subagent and the Step 3 contextual pass — loads every matching addendum file itself.

skills/done-check/SKILL.md:65

  • Auditor 2 concerns are required to quote the specific item clause because main context can’t open authoritative-text-rules item bodies; the same problem applies to auditor 1 for mechanical quality-list items (Step 2 prohibits main from loading those bodies). Without a required quote, main context can’t reliably triage or fix a ⚠ without violating the Step 2 prohibition.
   Return one verdict per item, from the three defined in Step 4 of
   <RULES_ROOT>/skills/done-check/SKILL.md; read that step's verdict
   sentence, which states what backs each of the three.

@ultimatile
ultimatile merged commit 8353aef into main Jul 28, 2026
1 check passed
@ultimatile
ultimatile deleted the feat/145-authoritative-text-rule branch July 28, 2026 14:21
ultimatile added a commit that referenced this pull request Jul 29, 2026
…ment,review-pipeline,review-pipeline-coderabbit): close the findings #146 left unfixed (#152)

* fix(done-check,todo-check,authoritative-text-rules,quality-list,implement,review-pipeline,review-pipeline-coderabbit): close the findings #146 left unfixed (#147)

Retire `<RULES_ROOT>` / `<QUALITY_LIST_ROOT>` for a single `<SKILLS_DIR>`,
resolved once in Step 0 of each runner and carried to every consumer, so a
marketplace / symlinked install resolves rule files from one place.

Extend the coverage check to both done-check auditors and make an unmatched
return halt without a table; require auditor rows to quote the item clause
they apply, since a purely-mechanical body stays closed to main context.

Define delta mode as a report narrowing over a caller-supplied baseline, with
the step 5 gate still binding the whole audit, and point the three callers at
that gate instead of restating its dispositions.

Mark the unbounded trigger enumerations in `consumer-closure` and
`executor-fitness` as illustrative, add their tie-breaks against
`single-reading` and `clause-composition`, and replace the stale `item 11`
reference with `paired-artifact-drift`.

* fix(done-check,review-pipeline,review-pipeline-coderabbit): state each halt and gate-scope rule once

Step 6 claimed a halted run "reports at all", contradicting the halt sites that
report the file looked for and the directory looked in. Narrow the claim to the
audit table, which is what the halts actually suppress.

The Output format section restated the halt-to-no-table rule instead of
referring to step 6, and both pipelines restated done-check's gate scope
instead of referring to its Delta mode section. Each condition now has one
statement and the other sites point at it.

* fix(done-check): retry an auditor row that omits its required clause

A concern row whose body main context may not open is judgeable only through
the clause the auditor quotes. Step 4 sent a row missing that clause straight
to the step 5 gate, where none of the three closures fits: main cannot derive
what to fix against, a deferral has nothing to track, and only a user waiver
remains. A missing slug in the same return already gets one re-dispatch.

Give the unquoted row the same allowance, with the replacing return standing
in for the first, and keep the existing disposition when the clause is still
absent. No new halt: the branch falls through to the gate it already had.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quality-list: no item owns text that is itself the agent's instructions

2 participants