Skip to content

test(action): dogfood CERTIFICATION_FAILED end-to-end - #327

Merged
BenSheridanEdwards merged 2 commits into
mainfrom
fix/dogfood-remaining-trust-states
Jul 21, 2026
Merged

test(action): dogfood CERTIFICATION_FAILED end-to-end#327
BenSheridanEdwards merged 2 commits into
mainfrom
fix/dogfood-remaining-trust-states

Conversation

@BenSheridanEdwards

Copy link
Copy Markdown
Owner

Why does this feature exist?

The action-dogfood workflow drives the composite Action against a fixture per trust state and asserts the verdict — it's the end-to-end safety net for the trust contract. It covered six of the eight states. The two gaps were CERTIFICATION_FAILED and REPORT_PUBLICATION_FAILED, and that first gap is not academic: 4.6.2's content-geometry regression wrongly routed ordinary copy-editing PRs into CERTIFICATION_FAILED (an unclearable state), and it shipped undetected precisely because nothing ever drove the Action into that state here. This closes it.

What changed?

  • scripts/action-dogfood-fixtures.mjs: a new certfail-base / certfail-head pair — identical maps, but a styleproof-coverage.json declaring determinism: 'unproven' (a side that was neither self-checked nor replayed). assessDeterminism turns that into status: 'unproven' (verified locally: styleproof-diff --json reports determinism.status: "unproven").
  • .github/workflows/action-dogfood.yml: a certfail step runs the Action against that pair and asserts it fails (outcome == 'failure') with trust-state == 'CERTIFICATION_FAILED' — proving unproven provenance certifies nothing and the approval box cannot clear it, with maps identical so provenance is the only thing under test.
  • test/action.test.mjs: the workflow-contract test now expects 8 Action invocations and pins the new certfail assertions.
  • REPORT_PUBLICATION_FAILED is left unit-tested (the trust step's publish/delivery failure domains are already pinned in action.test.mjs) but not end-to-end dogfooded: forcing a real publish failure needs a broken remote/permissions, which a self-contained dogfood can't stage cleanly. Called out in the commit as a follow-up rather than faked.

Behavioural Proof (with video and screenshots)

Not applicable — CI/Action trust-contract coverage with no rendered surface. The proof is executable: locally, node scripts/action-dogfood-fixtures.mjs && node bin/styleproof-diff.mjs action-dogfood/certfail-base action-dogfood/certfail-head --json reports determinism.status: "unproven"; the new dogfood step asserts the Action escalates that to a failing CERTIFICATION_FAILED; and this PR's own action-dogfood CI job exercises it.

Verification Summary

  • Full unit suite: npm test — 608/608 pass (updated the workflow-contract count 7 → 8 and added the certfail pins).
  • Local fixture proof: the certfail pair produces determinism.status: "unproven" through the real styleproof-diff.
  • Pre-commit gate green: tsc + typecheck, eslint, prettier, gitleaks, fallow, commitlint.
  • This PR's action-dogfood job is the live end-to-end proof (7 states now, was 6).

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

🗺️ StyleProof report

📊 View the side-by-side visual report →


Coverage, determinism, or report/diff consistency evidence is incomplete — repair the capture or reflow source; visual approval cannot clear this failure.

StyleProof Test and others added 2 commits July 21, 2026 17:38
action-dogfood exercised 6 of the 8 trust states; CERTIFICATION_FAILED and
REPORT_PUBLICATION_FAILED were the gaps. CERTIFICATION_FAILED is the one that
matters most — it's the state 4.6.2's content-geometry bug hid in, undetected
precisely because nothing drove the Action into it end-to-end.

A new certfail-base/head fixture pair carries identical maps but unproven
determinism (a side that was neither self-checked nor replayed). The diff
reports determinism.status='unproven' (verified locally), which the Action's
provenance gate must escalate to CERTIFICATION_FAILED — the state the approval
box cannot clear. The dogfood step asserts both the failure and the verdict;
action.test.mjs pins the workflow contract (8 action invocations now).

REPORT_PUBLICATION_FAILED remains unit-tested (the trust step's publish/delivery
domains are pinned in action.test.mjs) but not end-to-end dogfooded — forcing a
publish failure needs a broken remote, which a self-contained dogfood can't
stage cleanly; tracked as a follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts:270)

This console.warn — the tolerate-surface-failures capture-failure log — was the
one console.warn in runner.ts missing the // eslint-disable-next-line no-console
that its four siblings (623, 733, 947, 1031) all carry, so `eslint .`
(--max-warnings 0) failed the build. Bringing it in line with the file's pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@BenSheridanEdwards
BenSheridanEdwards force-pushed the fix/dogfood-remaining-trust-states branch from 42e478c to 24bea26 Compare July 21, 2026 16:38
@BenSheridanEdwards
BenSheridanEdwards merged commit 35a74dd into main Jul 21, 2026
12 checks passed
BenSheridanEdwards added a commit that referenced this pull request Jul 22, 2026
Cut 4.7.0 from the six commits merged since 4.6.3. Two are feature work, so
this is a minor bump; the version bump is the Release workflow's signal to
tag, publish to npm, and cut the GitHub Release on merge.

Added
- Reports publish through the git-data API (#328).
- styleproof-init prunes reports by API with a retention + size-budget sweep (#329).
- CERTIFICATION_FAILED dogfooded end-to-end (#327).

Security
- Cleared two high-severity transitive advisories: brace-expansion 5.0.6→5.0.7
  (#330) and fast-uri 3.1.3→3.1.4, GHSA-v2hh-gcrm-f6hx (#331).

Verified locally: build, typecheck, lint, format:check, privacy:check all
clean; full node --test suite 630/630 with a complete install.

Co-authored-by: StyleProof Test <styleproof@example.test>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant