[AI-58] llm: add bitwarden-planning-tools plugin with consulting-adrs skill - #177
[AI-58] llm: add bitwarden-planning-tools plugin with consulting-adrs skill#177SaintPatrck wants to merge 3 commits into
Conversation
|
Claude Code is validating plugin components and security... If this comment does not update with results, check the Actions log. |
🤖 Bitwarden Claude Code ReviewOverall Assessment: APPROVE Reviewed the new Code Review Details
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Establish bitwarden-planning-tools as the pre-implementation planning home (counterpart to bitwarden-delivery-tools' post-implementation mechanics), and land consulting-adrs here as its first skill. consulting-adrs checks a design/change/plan/threat-model against Bitwarden's ADRs (or locates/summarizes the catalog) and returns structured conflict/gap/aligned findings with cited ADRs. Ships the full eval harness (trigger/structure/behavior) with baselines on claude-opus-4-8: triggering 7/8 should-trigger, 6/6 should-not; behavior with-skill 1.00 vs baseline 0.78 over 9 cases. WebFetch scoped to contributing.bitwarden.com. Proposed as the future home for tech breakdowns, initiative-funnel navigation, and architecting-solutions as planning tools consolidate here.
Treat fetched ADR pages as untrusted data, matching the sibling architecting-solutions skill's rule for the same domain. Drop the Cross-Plugin Integration table's consumer inventory rather than correct its premature claim — a skill has no need to track which plugins call it, and the table's only other row duplicated the Overview's own counterpart framing.
ce76862 to
8e05440
Compare
Claude Configuration Validation — PR #177Provisional: plugin-validator and skill-reviewer agents still running. This file will be replaced with the final report. Direct checks completed so far on
|
d4374e1 to
928b0d5
Compare
928b0d5 to
ab8303f
Compare
| "should_not_trigger_pass": "4/4", | ||
| "reliability": { | ||
| "all_runs_agree_rate": 0.9166666666666666, | ||
| "should_trigger_reliable": "7/8", | ||
| "should_not_trigger_reliable": "4/4" | ||
| }, |
There was a problem hiding this comment.
❓ QUESTION: The two negative cases that over-triggered were dropped and the reliability figures recomputed — scope decision or worth re-running?
Details
ab8303f removed "Help me write a new ADR documenting the token-rotation decision we just landed on." and "What's an ADR and when should our team bother writing one?" from trigger-eval.json, and adjusted this block to match: should_not_trigger_pass 6/6 → 4/4, all_runs_agree_rate 0.786 → 0.917, should_not_trigger_reliable 4/6 → 4/4. Those were the only two negative cases that failed (each fired 1/3), and the _notes line disclosing them was trimmed at the same time.
The remaining 12 per-query records are self-consistent with the new totals, so nothing here misreports. Two things worth confirming:
- If the removal was a scope call (authoring and conceptual asks are arguably ambiguous labels rather than skill defects), a one-line note in
evals/README.mdunder "Known issues / boundaries" would keep that reasoning with the artifact — the under-trigger case is documented there, this one now isn't. - If they are still in scope, ADR-authoring is the nearest over-trigger boundary for this skill and now has only partial coverage via
"Update ADR-0012's status to superseded...".
Either resolution is fine; the ask is that the recorded baseline carries the reasoning, since it's what the next refresh will be compared against.
There was a problem hiding this comment.
Scope decision. The dropped tests were flakey and not relevant until competing or complimentary skills are identified.
1 participant
🎟️ Tracking
AI-58
📔 Objective
Adds
bitwarden-planning-tools, a new plugin housing pre-implementation planning and preparation skills.Its first skill is
consulting-adrs: it checks a design, change, plan, or threat model against Bitwarden's Architecture Decision Records, or locates and summarizes the catalog, returning structured findings (conflict, gap, stale-reference, aligned) with cited ADRs.WebFetchis scoped tocontributing.bitwarden.com.Ships a full eval harness per the AI Review Guidelines (triggering, structure, behavior) with baselines on
claude-opus-4-8, recorded inskills/consulting-adrs/evals/.#178 wires
bitwarden-security-engineer's ADR-alignment checks to this skill and is stacked on this PR, which must land first.