feat(evals): Tier 0 static lint + Tier 1 activation evals for all 18 skills

[jordanrburger]

Why

Skills have had no quality gate: descriptions drift, references go dead, and nobody measures whether a skill actually activates on the utterances it promises to catch. This is the ai-kit half of the four-tier skill eval loop designed with keboola/KaiBench (docs/skill-evals.md, landing in KaiBench#40): the fast tiers (0–1) live here and run on every PR; the heavy behavior/live tiers run from KaiBench.

Tier 0 — static lint (evals/lint/, pytest, seconds, no LLM)

Generalizes sl-toolkit's test_skill_consistency.py repo-wide:

• SKILL.md frontmatter parses as YAML; name is kebab-case and equals its directory; description present and ≤1024 chars
• command/agent frontmatter has a description (the dispatcher surface)
• marketplace.json ↔ plugin.json: sources exist, names match, versions match, every plugin on disk is published
• no dangling skill-internal references (references/, template/, assets/, examples/); relative markdown links resolve; skill scripts executable + BASH_SOURCE self-location per CLAUDE.md (with a justified exemption list)

Real drift the lint caught on first run, fixed in this PR:

• keboola-git SKILL.md frontmatter was invalid YAML (bare Triggers: colon inside a plain scalar)
• generate-vcr-tests command frontmatter was invalid YAML (unquoted argument-hint with two flow sequences)
• duckdb-transformation / keboola-config skill names didn't match their directories (DuckDB Transformation, Keboola Configuration)
• all 11 keboola-cli agents declared whenToUse: instead of description: — a field Claude Code doesn't read, making the agents invisible to the dispatcher (component-developer's agents, which demonstrably load, use description:)
• semantic-layer-usage pointed at references/ files that live in the sibling dataapp-development skill

Affected plugins patch-bumped per CLAUDE.md (component-developer 3.3.1, keboola-cli 1.1.1, keboola-git 1.0.1, dataapp-developer 1.5.1; marketplace 1.12.1).

Tier 1 — activation evals (evals/activation/)

Measures the routing precision/recall of every skill's description: field. The classifier (Claude Haiku, temperature 0) is shown the full marketplace skill list — the surface the real harness routes on — plus one labeled utterance, and asked which skills it would invoke; grading is deterministic (skill in answer iff should_trigger).

• Activation cases for all 18 skills (194 labeled utterances) at plugins/<plugin>/evals/<skill>/trigger-evals.json, generalizing the existing component-developer/get-started precedent. Negatives are hard: drawn from sibling skills' territory (debug vs develop vs get-started vs test…), not unrelated topics.
• Offline validation runs with no API key: every skill must have a case set; sets must parse, belong to a real skill, and contain ≥6 cases with both labels.
• --ci gates at ≥85% overall / ≥60% per-skill accuracy; summary.json records per-skill precision/recall and every misroute with the router's actual pick.

Baseline live run: 95.9% overall accuracy, recall 1.00 on all 18 skills. The 8 misroutes are co-activation false positives on deliberately borderline negatives — that list is the description-tuning backlog this loop exists to drive.

CI

.github/workflows/skill-evals.yml — this repo's first workflow. Tier 0 + offline case validation on every PR; Tier 1 runs when the ANTHROPIC_API_KEY secret is available (skips gracefully on forks) and uploads summary.json as an artifact.

Note for reviewers: the ANTHROPIC_API_KEY repo secret needs to be added for the Tier 1 job to run in CI; until then it self-skips.

Verification

• cd evals && uv run --group dev pytest -q → 323 passed (lint + case validation)
• Live activation run across all 194 cases → 95.9%, per-skill table + misroutes in the PR-linked summary

🤖 Generated with Claude Code

[jordanrburger]

Tier 1 is now live in CI. The ANTHROPIC_API_KEY repo secret was added and the workflow re-run on this PR (run 29687225641) — the activation job executed for real instead of self-skipping:

Job	Result	Duration
Tier 0: static lint + case validation	✅ pass	10s
Tier 1: skill activation routing	✅ pass	56s (previously 8s self-skip)

Tier 1 ran all 194 labeled utterances across the 18 skills through the Haiku router in CI:

18 skills, 194 labeled utterances
Overall accuracy: 95.9%
8 misroutes
VERDICT: PASS

Two things worth noting:

• 95.9% is identical to the local baseline run from the PR description — the temperature-0 classifier makes the gate reproducible, not flaky.
• The per-skill precision/recall table and the full misroute list are in the activation-summary artifact on the run. The 8 misroutes (all co-activation false positives on deliberately borderline negatives — recall is 1.00 on every skill) are the ready-made backlog for a first description:-tuning pass after this merges.

The reviewer note in the PR description about the missing secret is now resolved.

🤖 Generated with Claude Code

[jordanrburger]

Case set expanded with real-world utterances (88f99cb): mined first user messages from actual Kai conversations (the OTEL trace analysis tables — fct_otel_conversations.first_user_message in the Keboola AI project) and converted the recurring intent patterns into 45 new activation cases (194 → 239). Every utterance is an anonymized paraphrase — client names, IDs, URLs, and client-specific table names replaced; nothing copied verbatim.

What real traffic contributed that the synthetic set missed:

• Job-error debugging is the dominant real intent ("Resolve this job error…" deep-links) → new debug-component positives for output-mapping mismatches, OAuth grant failures, and Python tracebacks — plus the feat: Introduce company-wide prompt hub #1 real-world boundary as a hard negative: a SQL-transformation error is not Python-component debugging.
• The keboola-cli ↔ keboola-config transformation overlap shows up constantly in real asks (edit vs review vs schedule) — now captured as deliberate boundary cases on both sides.
• Real data-app lifecycle asks (create hello-world, preview dev version, slow-page optimization), a verbatim-in-spirit keboola-git ask ("use the keboola managed repo, I don't have a github account") with "set up GitHub integration for our flows" as its hard negative, and deprecated-component config migration as a hard negative for migrate-to-uv / develop-component / get-started alike.
• ~¼ of real Kai traffic is Czech/Slovak → added cs/sk utterances across skills, so the router is now graded on cross-language activation too.

The live run also produced two gold corrections (the router was right, my labels were wrong): converting an existing script or building a brand-new custom component is initial scaffolding → get-started, per develop-component's own boundary rule.

Result: 239 cases, 94.1% overall accuracy, VERDICT: PASS — all skills above the per-skill gate. The 14 misroutes are a sharper description-tuning backlog than before; the two standouts are the keboola-cli/keboola-config transformation-review split and "conditional flow" lexically attracting build-component-ui.

🤖 Generated with Claude Code

[vojtabiberle]

Four review notes from a local multi-dimensional review (3 minors + 1 nit). None block; see the APPROVE review for the overall verdict.

[vojtabiberle]

[minor] Per-skill CI gate on accuracy alone can pass a zero-recall skill

The --ci per-skill gate only checks accuracy < min_skill_accuracy (default 0.6). But recall is the metric Tier 1 exists to protect — "does the skill activate on the utterances it promises to catch". With the balance floor being MIN_POSITIVE=2 / MIN_NEGATIVE=2 over MIN_CASES=6, a set of 2 positives + 4 negatives that triggers on none of its positives (recall 0.0) still scores 4/6 = 66.7% > 60% and passes. A description that has gone completely dead would sail through the per-skill floor.

summarize() already computes precision/recall, so this is a few lines: add a per-skill recall floor (e.g. --min-skill-recall), or gate on min(precision, recall).

[vojtabiberle]

[minor] Skill lookup keyed by bare dir-name — cross-plugin collision would mis-map

by_dir here (and SKILLS_BY_DIR in test_activation_cases.py) is keyed on dir_name only. If two plugins ever ship a skill directory with the same name (e.g. both a review/), the dict silently keeps the last one, and a case set for the other plugin's same-named skill either fails the plugin assertion or gets attributed to the wrong plugin in the summary. No collision exists today, so this is latent — but the design assumes a global dir-name uniqueness that nothing enforces.

Skill.qualified (f"{plugin}:{dir_name}") already exists as the natural composite key — suggest keying both maps on (plugin, dir_name) and looking up case sets by the same.

[vojtabiberle]

[minor] parse_answer takes the first bracket pair — stray brackets grade as a silent false-negative

The lazy \[.*?\] matches the first [...] in the reply. If the classifier ever emits prose containing brackets before the JSON (e.g. Based on [the description] I'd pick ["x"]), the regex captures [the description], json.loads fails, and parse_answer returns [] — scored as "invoked nothing", i.e. a false negative in grading rather than a visible error. Risk is low at temperature 0 with the strict "Reply with ONLY a JSON array" prompt, but it fails silently.

Suggest preferring the last bracket match (or stripping a ```json fence), and on JSONDecodeError recording the raw reply in the result row so a parse failure is distinguishable from a genuine empty answer.

[vojtabiberle]

[nit] tier1-activation doesn't depend on tier0-lint — a red lint still burns a paid API run

The two jobs run in parallel, so if Tier 0 (the cheap, deterministic gate) fails, Tier 1 still fires ~194 Haiku calls on an already-failing PR. Adding needs: tier0-lint to this job means the paid tier only runs once the free tier is green.

[vojtabiberle]

RECOMMEND APPROVE WITH NITS — no blockers, no majors.

Adds a self-contained evals/ harness (pytest + uv): Tier 0 static lint over every SKILL.md / command / agent / manifest, and Tier 1 an LLM-classifier activation eval measuring each skill description:'s routing precision/recall against 194 labeled utterances. Ships the repo's first CI workflow and fixes real pre-existing drift (invalid YAML frontmatter, name↔dir mismatches, 11 agents using an unread whenToUse: field). Well-scoped, self-contained, tests green (323 offline + 95.9% live baseline).

Left 3 minors + 1 nit as inline comments — none block. The one worth prioritizing is the per-skill CI gate keying on accuracy alone, which can pass a zero-recall skill; that undercuts the harness's stated purpose but is a small fix (recall is already computed). The fork-secret handling in the workflow is correct (pull_request, not pull_request_target).

Security reviewed inline; connection/platform-impact dimension is N/A (different repo, no cross-service contract touched).

[jordanrburger]

Thanks @vojtabiberle — all four addressed in 3d79875, CI green on the new run (330 offline tests + live Tier 1 at 94.1% PASS):

• Zero-recall skill passing the gate — --ci now also enforces a per-skill recall floor (--min-skill-recall, default 0.6), with the failure message spelling out what it means ("description not activating on its own positives"). Your 2-pos/4-neg dead-description scenario now fails the gate exactly as it should.
• Cross-plugin dir-name collision — both the runner and the offline case validation now key skill lookups on (plugin, dir_name); a case set must name a skill within its own plugin. The redundant plugin assertion in the offline test collapsed into the composite-key check.
• parse_answer first-bracket grab — now prefers a ```json fence, then tries bracket candidates last-first, and an unparseable reply returns the raw text: it's recorded as parse_error on the result row and flagged `[UNPARSEABLE REPLY]` in the misroute listing, so a parse failure is never silently graded as "invoked nothing". New offline unit tests (`test_parse_answer.py`) cover the prose-brackets-before-JSON case, the fence, and the error path.
• Paid tier on red lint — tier1-activation now has needs: tier0-lint; visible on the new run, where Tier 1 queued until lint finished.

🤖 Generated with Claude Code