Skip to content

feat(agent-context): add metaobjects-audit adoption-review skill#93

Merged
dmealing merged 12 commits into
mainfrom
feat/metaobjects-audit-skill
Jun 29, 2026
Merged

feat(agent-context): add metaobjects-audit adoption-review skill#93
dmealing merged 12 commits into
mainfrom
feat/metaobjects-audit-skill

Conversation

@dmealing

Copy link
Copy Markdown
Member

Intent

Build a new 'metaobjects-audit' agent-context skill (the 6th metaobjects-* skill) that reviews how well an adopter project has integrated MetaObjects and where it can improve — supporting both brand-new (greenfield first-pass) and deeply-integrated projects, covering codegen AND runtime-driven usage. Design decisions: the skill is read-only/propose-only (never mutates adopter code), emits two artifacts (.metaobjects/adoption-audit.json typed findings + a Markdown report), and uses a THREE-surface scoring model with NO single global score (a maturity tier Greenfield->Partial->Deep->Exemplary that is AND-gated/worst-of, a per-pillar breakdown, and a binary CI drift-gate signal). It has an audit->action bridge (typed findings -> dry-run -> review -> apply, mapping onto 'meta gen --dry-run' + 'verify --codegen'). Capability coverage is registry-grounded: references/capability-checklist.md enumerates every metamodel type/subtype/attr and a grounding test asserts each token exists in expected-registry.json with principled cut/TS-only/planned exemptions. The skill DELIBERATELY names retired/cut anti-pattern vocabulary (source.dbTable, @dbColumn, field.byte/short/class) because its job is to recognize them, so it is exempted from the cruder drift.test.ts vocabulary scan and governed instead by the stricter grounding test (which the fix wave extended to scan SKILL.md + all references, not just the checklist). Five per-port reference fragments (ts/csharp/java/kotlin/python) carry real CLI commands + calibration rules that mark per-port codegen gaps and cross-port version skew as NON-defects. Registered as 6th/last in SKILL_NAMES; all 4 agent-context conformance fixtures regenerated. Executed via subagent-driven-development with a final whole-branch review + one fix wave that removed an invented 'meta gen --watch' command (also removing the same dropped line from the sibling metaobjects-codegen skill), closed a grounding coverage hole, and reworded dangling spec cross-references. Includes the design spec + implementation plan docs.

What Changed

  • Adds a new metaobjects-audit agent-context skill (6th and last in SKILL_NAMES) — a read-only/propose-only adoption reviewer that emits typed .metaobjects/adoption-audit.json findings plus a Markdown report, scores integration via a three-surface model (AND-gated maturity tier, per-pillar breakdown, binary CI drift-gate) with no single global score, and bridges audit→action onto meta gen --dry-run + verify --codegen. Ships SKILL.md, a registry-grounded capability-checklist.md, and five per-port reference fragments (ts/csharp/java/kotlin/python) with real per-port CLI commands and calibration rules treating per-port codegen gaps and version skew as non-defects.
  • Adds an agent-context-capability-grounding.test.ts that asserts every metamodel token in the checklist (extended in the fix wave to all of SKILL.md + references) exists in expected-registry.json, with principled cut/TS-only/planned exemptions; exempts the audit skill from the cruder drift.test.ts vocabulary scan since it deliberately names retired anti-pattern vocabulary (source.dbTable, @dbColumn, field.byte/short/class) to recognize them.
  • Regenerates all four agent-context conformance fixtures (csharp/java-react/python/typescript-react-tanstack) to bundle the new skill, removes the invented meta gen --watch command from both the audit and sibling metaobjects-codegen skills, and adds the design spec + implementation plan docs.

Risk Assessment

✅ Low: The change is almost entirely additive documentation (a new read-only skill, design/plan docs, regenerated fixtures) plus a one-line constant addition and well-constructed tests whose registry-grounding assertions I independently verified pass; no behavioral code paths are altered.

Testing

Ran the change's three new/modified tests (registry-grounding, types/SKILL_NAMES, drift exemption) plus the agent-context conformance corpus and the full sdk agent-context suite — all pass. Because this is an inspectable-Markdown agent-context skill with no executable runtime, I demonstrated the real end-user surface by driving assemble() for a TS+React adopter stack and capturing the 7 audit files that ship into .claude/skills/metaobjects-audit/ along with the rendered SKILL.md the agent reads; the conformance test independently proves that assembly is byte-identical to the committed fixtures across all four stacks. One initial failure in bundle.test.ts was purely environmental (the gitignored build bundle hadn't been generated in this fresh checkout); running the bundle script produced it correctly including the new skill, after which I removed the transient output, leaving the worktree clean.

Evidence: Audit skill assembled into an adopter repo (file list + rendered SKILL.md head)
Files shipped into adopter repo for the metaobjects-audit skill:

  .claude/skills/metaobjects-audit/SKILL.md
  .claude/skills/metaobjects-audit/references/capability-checklist.md
  .claude/skills/metaobjects-audit/references/csharp.md
  .claude/skills/metaobjects-audit/references/java.md
  .claude/skills/metaobjects-audit/references/kotlin.md
  .claude/skills/metaobjects-audit/references/python.md
  .claude/skills/metaobjects-audit/references/typescript.md

--- metaobjects-audit/SKILL.md (first 40 lines as the agent would read it) ---

---
name: metaobjects-audit
description: Use when assessing how well a project has adopted MetaObjects — greenfield first-pass or deep double-check; produces a scored, prioritized adoption-audit report covering codegen, runtime, drift-gates, and prompts.
---

# MetaObjects adoption audit

**Thesis.** Typed metadata is the durable spine; generated code is the disposable
artifact. Hand-writing a layer the metadata could own creates a second source of
truth for one fact — it will drift. This audit hunts those second sources of truth
and proposes folding them into the spine.

**Boundary — read-only.** Deliverables: `.metaobjects/adoption-audit.json` (machine-readable
findings) + a rendered Markdown report. The audit **never edits code, never authors
metadata** — `metadata_sketch` per finding is a read-only proposal for human review.
Actual cutovers run through the existing skills mapped per finding tier (§ Bridge).

---

## Phase 0 — Triage (fast, mechanical)

- [ ] MetaObjects present? (`metaobjects/` dir, metadata sources, `@metaobjectsdev/*` /
  `com.metaobjects:*` / `metaobjects` / `MetaObjects.*` deps).
- [ ] Count metadata source lines + all `@generated` / `DO NOT EDIT` files repo-wide.
- [ ] **Owned-generators check:** does the project own generators at `codegen/generators/*`
  (scaffold-and-own via `meta init`), or still import the **deprecated** package export
  (`@metaobjectsdev/codegen-ts/generators`)? Not owning is itself a finding.
- [ ] Classify: **Greenfield** (none/minimal) · **Partial** · **Deep** → choose path below.

---

## Phase 1a — Greenfield path

- [ ] **Shape inventory.** Catalog modelable shapes: entities/tables, DTOs, validation
  schemas, routes, UI lists/forms, prompt sites.
- [ ] **Pick wedge:** one real entity (single-column PK, standard CRUD) to model first.
- [ ] **From-zero roadmap:** `meta init` → model the wedge → `meta gen` the data layer →
  author a projection view → expand to routes/UI → add prompt pillar where LLM calls exist.
  Owning the generators from day 1 is part of the roadmap.
Evidence: Grounding/conformance/registration test transcript
bun test v1.3.8 (b64edcb4)

 11 pass
 0 fail
 138 expect() calls
Ran 11 tests across 3 files. [321.00ms]

Test files exercised (all pass, 0 fail):
  agent-context-capability-grounding.test.ts
    - capability checklist is registry-grounded > every type.subtype named exists in the registry
    - capability checklist is registry-grounded > every @attr named exists in the registry
    - audit skill files (SKILL.md + references) are registry-grounded > there are skill files to check
    - audit skill files (SKILL.md + references) are registry-grounded > every type.subtype named exists or is a principled exemption
    - audit skill files (SKILL.md + references) are registry-grounded > every @attr named exists or is a principled exemption
  agent-context-conformance.test.ts  (assemble() == committed adopter fixtures, byte-identical)
    - csharp, java-react, python, typescript-react-tanstack
  agent-context/types.test.ts
    - the six skills are named and ordered  (metaobjects-audit registered 6th/last)
  agent-context/drift.test.ts
    - vocabulary-drift scan excludes /skills/metaobjects-audit/ by design; rest of corpus clean

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ server/typescript/packages/sdk/test/agent-context-capability-grounding.test.ts:95 - In the checklist @attr grounding test, the comment 'Allow doc-only/config attrs not in the metamodel registry (apiPrefix etc.)' promises tolerance for non-metamodel attrs, but the assertion expect(unknown).toEqual([]) provides no exemption set (unlike the audit-skill-files block, which has CODE_IDENTIFIERS_NOT_METAMODEL_ATTRS). It passes today only because such attrs aren't written backtick-wrapped with an @ in the checklist. A future editor adding e.g. @apiPrefix to the checklist would get a failing test the comment says should be allowed. Harmless now; the comment just contradicts the code.
✅ **Test** - passed

✅ No issues found.

  • bun test packages/sdk/test/agent-context-capability-grounding.test.ts — registry-grounding of SKILL.md + all references against expected-registry.json
  • bun test packages/sdk/test/agent-context-conformance.test.ts — assemble() == committed adopter fixtures byte-identical for csharp/java-react/python/typescript-react-tanstack (skill shipped into .claude/skills/metaobjects-audit/)
  • bun test packages/sdk/test/agent-context/types.test.ts — metaobjects-audit registered 6th/last in SKILL_NAMES
  • bun test packages/sdk/test/agent-context/drift.test.ts — audit skill excluded from vocabulary-drift scan, rest of corpus clean
  • bun test packages/sdk/test/agent-context (full suite, 42 tests)
  • Drove assemble() for a TS+React adopter stack and confirmed all 7 audit files land under .claude/skills/metaobjects-audit/ with correct SKILL.md content (evidence transcript)
  • node packages/sdk/scripts/bundle-agent-context.mjs then bundle.test.ts — fixed env-only failure (ungenerated build bundle), confirmed audit skill bundles correctly, then removed transient output
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

dmealing and others added 12 commits June 29, 2026 00:40
A 6th metaobjects-* skill that audits any project for MetaObjects adoption across
the full spectrum (greenfield first-pass → deep double-check) and both output modes
(codegen + dynamic-runtime), including the prompt pillar and the scaffold-and-own
owned-generator model. Read-only; deliverable is a tiered-roadmap report file.

Triage (greenfield/partial/deep) → adaptive methodology (census, coverage matrix,
surface-review axes incl. owned-generators, drift hunt) → classification (adds
OWNED-GENERATOR + dual codegen/runtime fit) → semantic-constraint ratification →
report. Folds in the brief's hard-won gotchas as guardrails; blends ADR-0034
scaffold-and-own + declarative template-codegen as first-class. Ships via
SKILL_NAMES + agent-context conformance + per-port references.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
Folded in a 3-agent surface-area sweep (full metamodel vocabulary + all docs/ADRs +
the 5 existing skills) so the audit misses nothing:

- Expanded Phase-3 to 8 axes spanning ALL FOUR pillars + authoring-correctness:
  added F (drift-gate adoption — is `verify` wired into CI at all?), G (runtime-
  contract anti-patterns: ADR-0008/0019/0017/0014/no-mutate/startup-validator),
  H (authoring/ADR-conformance: strict provenance, source-v2, taxonomy purity,
  YAML coercion, canonical FQN refs, Node-only migrations).
- New §4b + references/capability-checklist.md: the exhaustive modelable-capability
  hunt list derived from expected-registry.json (every object/field/source/
  relationship/identity/origin/validator/view/layout/template/attr capability).
- New §10b calibration guards (the audit's worst failure = flagging a by-design
  port gap): filter-op codegen TS-only, parser-not-Java, Python hand-wired router,
  C# no-ObjectManager, cut field.byte/short/class, TS-only view.* widgets,
  planned-not-shipped api.*/MCP, cross-port version skew is by-design.
- Prompt §7: added origin.collection, template.toolcall, @responseRef + AI-trace,
  @maxChars/@maxtokens, byte-stable-render. §8: cross-field validators ARE modelable.
- §11: capability-checklist always-on; per-port refs carry that port's calibration
  gaps; SKILL maps each cutover to the right sibling skill; checklist diffed against
  the registry manifest (content-review rule).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
Researched scoring/grading + audit→action against Lighthouse, SonarQube,
CodeClimate, OpenSSF Scorecard, Backstage-Soundcheck, CNCF, OpenRewrite, codemods
(ts-migrate / ng update), Renovate/Snyk, and the cloud migration assessors. Added:

- §8b Scoring & maturity model: deliberately NO single global score (the Scorecard
  mistake). Three surfaces — headline maturity TIER (Greenfield→Partial→Deep→
  Exemplary, worst-of/AND-gated) + per-pillar breakdown (codegen/runtime/drift/
  prompts, never averaged) + one binary CI drift-gate (SonarQube quality-gate analog).
  Bands not decimals; grade-the-delta on re-run; lead with gaps not grade.
- §9/§9b: two artifacts (.metaobjects/adoption-audit.json + Markdown), report leads
  with the scorecard; findings are TYPED RECORDS (id/pillar/surface/capability/
  locations/impact/effort/confidence/metadata_sketch/next_command/parity_gate/tier),
  tier-then-impact÷effort prioritized.
- The audit→action bridge: dry-run→review-diff→apply mapped onto meta gen --dry-run
  + meta verify --codegen; per-finding next_command routes to existing skills;
  parity_gate per finding; an optional guided-cutover follow-on skill; future
  auto-fix codemods only for the mechanical dead-code/import half (authoring stays
  human-reviewed). metadata_sketch is a proposal only — never authored/applied.
- §1/§12 updated for the artifacts + deferred follow-ons.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
5-task plan: author SKILL.md (methodology + scoring + report + audit→action
bridge) + a registry-grounded capability checklist (with an executable
grounding test so it can't claim vocabulary the registry lacks) + 5 per-port
reference fragments with calibration; register in SKILL_NAMES; nudge the
always-on template; regenerate the byte-gated agent-context conformance
fixtures; content-review for invented APIs + final verification.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
Drop the non-existent -Dmeta.gen.list Maven flag (Java/Kotlin); fix C# gen to
positional <metadataDir> --out (no --metadata-dir/--output-dir); correct C#/Python
agent-docs to the ADR-0033 npx meta agent-docs redirect form.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
…conformance

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
… scan

The drift test forbids agent-context from naming retired/cut/fake vocabulary —
correct for the teaching skills, but the audit skill's job is to RECOGNIZE
anti-patterns, so it deliberately names retired forms it hunts (source.dbTable,
@dbColumn) and the cut stubs it warns against (field.byte/short/class). The audit
skill's real-vocabulary claims are gated instead by the stricter, registry-grounded
agent-context-capability-grounding test. Scoped exemption + rationale comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
…le skill (review)

Final whole-branch review findings:
- Remove the dropped `meta gen --watch` command from the TS reference + regen the 4 conformance fixtures (was the invented-command class Task 3 targeted). Also removed from the metaobjects-codegen skill's typescript.md which shipped the same line into fixtures.
- Extend agent-context-capability-grounding to scan SKILL.md + references/*.md (not just capability-checklist.md), with a documented exemption for the retired/cut tokens the audit skill names on purpose; make drift.test.ts's "gated by grounding" comment accurate.
- Widen the grounding subtype regex to include api|operation|binding so the planned-token exemptions become a live guard.
- Reword dangling design-spec section cross-references in the shipped SKILL.md + capability-checklist.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LuZWKnWzYGVnESijL7uuky
@dmealing
dmealing merged commit 4e262b3 into main Jun 29, 2026
25 of 27 checks passed
@dmealing
dmealing deleted the feat/metaobjects-audit-skill branch June 29, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant