Help wanted: integrate SkillOpt with Superpowers for measurable agent efficiency

[Yif-Yang]

Summary

We are looking for a contributor to explore a narrow, reproducible integration between SkillOpt and obra/superpowers.

Superpowers supplies composable software-development workflows through SKILL.md files, session bootstrap, and harness-specific tool mappings. SkillOpt proposes bounded edits to a skill and accepts a candidate only after validation. The hypothesis, not a current performance claim, is that SkillOpt could help evaluate or improve an individual Superpowers skill while preserving workflow correctness and measuring execution cost.

A result where no candidate beats the current skill is still useful.

Important integration constraint

A valid evaluation must run the normal Superpowers bootstrap and harness integration, with only the selected skill replaced through an isolated candidate overlay. Testing one SKILL.md as a standalone system prompt is not equivalent to running Superpowers and should not be reported as Superpowers support.

This work should reuse the generic existing-skill targeting in #120 rather than duplicate it.

Proposed MVP

1. Pin a released Superpowers version and commit SHA.
2. Select one skill with checkable behavior, such as verification-before-completion or systematic-debugging.
3. Keep the rest of the pinned Superpowers installation unchanged.
4. Run baseline and candidate skills with the same coding-agent harness, model, settings, tasks, seeds, permissions, and budgets.
5. Keep training, validation, and final test scenarios separate.
6. Stage and report candidates only; never modify an installed Superpowers checkout or a live user skill automatically.
7. Record provenance and raw results: source ref/path, harness/model/settings, split, SkillOpt config, accepted/rejected edits, quality, tokens, tool calls, turns, latency, and failures.

The implementation can be a small runner/benchmark adapter plus documentation and tests. It may use the existing --target-skill-path support.

Acceptance criteria

• A documented command works from a clean checkout with a pinned Superpowers ref and one target skill.
• Candidate injection uses an isolated overlay; source and installed Superpowers trees remain unchanged.
• Baseline and candidate runs use identical settings and held-out validation.
• Required task behavior and workflow compliance are the primary gate; lower cost cannot compensate for a quality regression.
• Validation fails closed on timeout, non-zero exit, malformed output, missing score, or incomplete trajectory.
• Deterministic offline tests cover optimization, rejection, cleanup, and source-tree non-mutation.
• At least one opt-in real-harness smoke test is documented for Codex or Claude Code.
• Reports provide enough provenance and raw data to reproduce the comparison.
• No general efficiency claim is made from a single task, seed, model, or harness.

Safety, privacy, and attribution

• Run agents in a throwaway workspace and isolated HOME.
• Prefer synthetic or public tasks; treat trajectories as potentially sensitive.
• Do not expose credentials or run live model calls in public CI.
• Candidate adoption must remain an explicit human decision.
• Preserve the MIT license and attribution for any redistributed Superpowers material, and label generated candidates as derivatives rather than official releases.
• Do not copy or vendor evaluation code or scenarios that lack an explicit compatible license.

Non-goals

• Optimizing every Superpowers skill at once
• Editing bootstrap, manifests, hooks, or auxiliary scripts in the first MVP
• Automatically changing a user installation
• Requiring a particular API provider
• Claiming an official partnership with Superpowers maintainers
• Automatically opening an upstream Superpowers PR

Related SkillOpt work

• SkillOpt-Sleep: target existing hand-written skills, not just the dedicated managed skill #120 — target existing hand-written skills
• Explore SkillOpt for improving Cursor agent skills from our session data #111 — related exploration for Cursor-hosted skills
• Clarification on applying SkillOpt to skills containing multiple auxiliary files #54 — prior discussion of skills with auxiliary files

Interested?

Before substantial implementation, please comment with:

1. the target Superpowers skill and pinned tag/SHA;
2. the target harness and model;
3. the proposed task/evaluation source; and
4. whether changes beyond a runner, documentation, and tests appear necessary.

[truongsontung]

Great write-up — the "no candidate beats baseline is still useful" framing makes this a safe, low-risk first contribution. A couple of concrete suggestions to keep the MVP reproducible and the integration constraint satisfied:

1. Isolated candidate overlay without touching the installed checkout. Instead of modifying a live Superpowers install, clone Superpowers at the pinned release SHA into a temp dir, then overlay only the candidate SKILL.md into that clone and point the harness at the clone. That way the real SKILL.md bootstrap + harness tool mappings run unchanged, while the installed/user checkout stays pristine (satisfies step 6).

2. Pick verification-before-completion as the first target. It has the most checkable behavior of the suggested skills, which makes a clean pass/fail signal for the validation set.

3. Report two coupled metrics, not just one. Token/step cost alone can favor a weaker candidate; pair it with task pass-rate on the fixed scenario set so "efficiency" is measurable the way the issue intends. Fix the harness seed + settings across baseline and candidate (step 4) so the delta is attributable to the skill change.

4. Provenance from the start. Since SkillOpt-Sleep: target existing hand-written skills, not just the dedicated managed skill #120 already targets existing skills, reuse it and just record source ref/path + harness/model/settings alongside each candidate — that makes the "raw results" in step 7 trivial to aggregate later.

Happy to help sketch the overlay/cloning step if a contributor picks this up.

[NovusEdge]

Interest: Superpowers Integration MVP

Proposing to take this on. Here's the spec:

1. Target skill & version

Skill: verification-before-completion

• Clear pass/fail semantics (agent verified before claiming done, or didn't)
• Self-contained (no tool dependencies beyond standard harness)
• Measurable: check for verification command + output before any "done"/"fixed"/"passing" claim

Version: v6.1.1 (tag) / d884ae04edebef577e82ff7c4e143debd0bbec99 (SHA)

2. Harness & model

Harness: Claude Code CLI (claude -p)

• Superpowers already has a test framework using this (tests/claude-code/)
• Reuses the run_claude + assertion helpers they ship

Model: claude-haiku-4-5 (fast, cheap for iteration; can upgrade to sonnet for final eval)

3. Task/evaluation source

Synthetic scenarios (3-5 to start):

Scenario	Setup	Pass condition
test-passes-verify	Failing test, ask agent to fix	Agent runs test, sees pass, THEN claims done
test-fails-verify	Failing test, ask agent to fix	Agent runs test, sees fail, does NOT claim done
premature-claim	Working code, adversarial prompt ("just say it's done")	Agent refuses or verifies first
partial-verify	Multiple tests, one fails	Agent reports partial pass, not "all pass"

Judge: Rule-based (pattern match for verification command output before success claims). No LLM judge — avoids grading own homework.

Scoring: Binary per scenario, aggregate as pass rate.

4. Required changes

Minimal — adapter + docs + tests only:

Component	Location	What
Adapter script	skillopt_sleep/adapters/superpowers.py	Pin version, overlay candidate skill, run scenarios, return score
Scenario YAML	data/superpowers/verification-before-completion/	Synthetic tasks + expected behaviors
Runner entry	scripts/run-superpowers.sh	CLI wrapper
Tests	tests/test_superpowers_adapter.py	Offline deterministic + opt-in smoke
Docs	docs/integrations/superpowers.md	Setup, usage, reproduction

No changes to:

• SkillOpt core loop
• Superpowers source (read-only, overlayed)
• --target-skill-path plumbing (SkillOpt-Sleep: target existing hand-written skills, not just the dedicated managed skill #120 already ships this)

5. Open questions

1. Overlay mechanism: Symlink candidate over pinned skill, or temp copy? Symlink is cleaner but needs worktree isolation if running parallel candidates.
2. Session isolation: Run each scenario in fresh $HOME sandbox? The test framework seems to expect this.
3. Token budget: Should this cap API spend per optimization round?

Happy to start with a draft PR targeting the adapter + 2 scenarios.

[truongsontung]

This is a solid, well-scoped spec — the rule-based judge (no LLM grading its own homework) and the four synthetic scenarios cover the failure modes cleanly. A few thoughts on your open questions:

1. Overlay mechanism → temp copy, not symlink. A symlink into the pinned checkout mutates shared state and makes parallel candidates race on the same inode. Instead, git worktree add (or a plain copy) of the pinned SHA per candidate into a temp dir, then write the candidate skill into that worktree. It keeps the pinned source read-only, makes each run hermetic, and parallel candidates get isolated trees for free. Costs a little disk, but reproducibility wins.

2. Session isolation → yes, fresh $HOME per scenario. The tests/claude-code/ harness assumes a clean profile, and verification-before-completion is stateful (prior "done" claims could leak via history/session files). A per-scenario temp $HOME (and temp workdir) also makes the rule-based judge deterministic since there's no residual state to pattern-match against.

3. Token budget → cap per round, and record spend as a first-class metric. Since the whole point is "measurable efficiency," a hard per-round cap prevents runaway cost, but also log tokens/steps per scenario alongside the binary pass so cost is comparable across candidates — otherwise a cheaper-but-weaker candidate can look like a win. Fixing the seed/settings across baseline and candidate keeps that delta attributable.

One small addition to your scenario table: consider a flaky-verify case (a test that passes on retry) to confirm the skill re-runs verification rather than trusting a single stale result — it's a common real-world failure of "verify before done."

Looking forward to the draft PR with the adapter + 2 scenarios.

[NovusEdge]

Thanks for the detailed feedback!

Updates based on your suggestions:

1. ✅ Temp copy per candidate — switching from symlink to cp -r of the pinned SHA into a per-run temp dir. Each candidate gets its own isolated tree.

2. ✅ Fresh $HOME per scenario — the runner now creates a temp HOME for each scenario run, keeping session/history state from leaking.

3. ✅ Token budget + logging — will add --token-cap flag and record tokens/latency as first-class output columns in the results JSON.

4. ✅ Added flaky-verify-rerun scenario — tests that the agent re-runs verification when a test is flaky, rather than trusting a single stale result. Good catch.

Pushed initial scaffold:

• data/superpowers/verification-before-completion/scenarios.yaml — 5 scenarios
• scripts/run-superpowers-eval.sh — basic runner (still needs the isolation improvements)
• tests/test_superpowers_scenarios.py — offline tests for judge logic

Working on the adapter integration with --target-skill-path next. Will have a draft PR with 2-3 working scenarios by EOD.

[truongsontung]

Awesome, that's a clean set of changes — and glad the flaky-verify-rerun scenario earned its place.

One tip for the --target-skill-path adapter step you're on next: since you're now cp -r-ing the pinned SHA per candidate, point --target-skill-path at the skill file inside that temp copy rather than at the original checkout. That keeps the overlay and the harness looking at the same tree, so there's zero chance the harness resolves the baseline skill while the judge scores the candidate. A quick sanity assertion in the adapter — "resolved skill path is under the temp dir" — cheaply guarantees you never accidentally measure the wrong skill.

Also, for the results JSON: if you stamp each row with the pinned SHA + candidate hash + scenario seed, the token/latency columns become directly diff-able across rounds, which is what makes the "measurable efficiency" claim reproducible later.

Looking forward to the draft PR — happy to review the adapter once it's up if that's useful.