Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

Format: [Keep a Changelog](https://keepachangelog.com). Versioning: semver — for skills *and* for this CLI, breaking prompt changes are breaking changes.

## [0.13.0] — 2026-07-29

### Fixed
- **Cline was being charged a standing tax it stopped owing, and served the same skill twice.** The `cline` adapter compiled to `.clinerules/<name>.md` — a *rule*, always active, whole body in context every session — and declared the target eager. Cline reads Agent Skills (`apps/vscode/src/core/storage/skill-directories.ts` scans `.cline/skills`, `.clinerules/skills`, `.claude/skills` and `.agents/skills`) and loads them progressively: name and description at startup, the body only when `use_skill` fires. So the target was billed ~539 standing tokens for a skill Cline was willing to lazy-load for ~40. Worse, in any repo that also had `.agents/`, Kitbash emitted the identical body to both paths and Cline scanned both — the always-on copy defeating the lazy one, and a tool whose entire pitch is measuring standing cost silently adding ~539 tokens of it. `cline` now compiles to `.agents/skills/<name>/SKILL.md`, the same vendor-neutral file `agents` and `zed` emit, which makes the duplicate structurally impossible: one file serves Cline, Codex, Cursor, Copilot and Zed at once. Detection gains `.cline/`. Stale `.clinerules/<name>.md` output is pruned on the next compile; your own rule files there are untouched.
- **The measured tax moved with it**, because the numbers are read from the adapters rather than restated: **14×** for a manifested skill (~40 lazy vs ~560 eager) and **47×** for an unmanifested one, up from 13× and 45×. Cline leaving the eager set costs a row in a table, not the argument — `aider` and `agentsmd` still carry the whole body every session. Eight of the ten targets are now lazy.

### Added
- **A skill authored `disclosure = "eager"` now warns on `cline`.** Cline loads every skill on demand, so an eager-authored skill does not get the always-resident body it asked for. Compiling that silently would fix one spec §2 conformance violation by introducing its mirror image; it warns instead, and fails `--strict`.

## [0.12.0] — 2026-07-28

### Added
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ If you've used npm for packages, Docker for containers, or ESLint for lint rules

**Stable specification, experimental ecosystem.** The Kitbash Skill Format (KSF) core is stabilized through [RFC 0002](rfcs/0002-ksf-1.0-stabilization.md): the manifest fields are frozen and evolve additive-only within the major version, so you can author skills and write adapters against a contract that won't shift under you. The ecosystem around it — more adapters, the index, first-party skills — is still early.

> **Compiler insight** — Kitbash measures a skill's *standing token cost* (what it adds to your context every session) at compile time, before you ever install it. For prereview that's ~40 tokens on a lazy target; compiled to an eager one it's ~540 — a [13× per-session tax](docs/benchmarks/README.md) no other format surfaces. Run `npm run bench` for the numbers.
> **Compiler insight** — Kitbash measures a skill's *standing token cost* (what it adds to your context every session) at compile time, before you ever install it. For prereview that's ~40 tokens on a lazy target; compiled to an eager one it's ~560 — a [14× per-session tax](docs/benchmarks/README.md) no other format surfaces. Run `npm run bench` for the numbers.

## A quick look

Expand All @@ -29,7 +29,7 @@ If you've used npm for packages, Docker for containers, or ESLint for lint rules

That's an actual session. A third-party skill from the [skills.sh](https://www.skills.sh) convention gets installed and compiled into three agent formats. The thing to notice is the last warning: during compile, Kitbash measured the skill and pointed out that it quietly costs about 5,044 tokens on every request for agents that can't lazy-load. A converter would just translate the format. The compiler reads it and tells you what it's going to cost you. I haven't found another tool that surfaces that number.

That gap is measured, not asserted — see the [benchmark](docs/benchmarks/README.md). Kitbash compiles to the cheapest loading mode each target actually supports, so prereview costs ~40 standing tokens on a lazy target; the tax is what it costs on the targets whose only mode is eager — ~540 tokens, a 13× per-session gap that a team running four agents pays four times over. Reproduce it with `npm run bench`.
That gap is measured, not asserted — see the [benchmark](docs/benchmarks/README.md). Kitbash compiles to the cheapest loading mode each target actually supports, so prereview costs ~40 standing tokens on a lazy target; the tax is what it costs on the targets whose only mode is eager — ~560 tokens, a 14× per-session gap that a team running four agents pays four times over. Reproduce it with `npm run bench`.

Install it (npm or Homebrew — see [Installation](#installation)), then in a repo:

Expand Down
8 changes: 4 additions & 4 deletions docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ Token counts are estimates (~4 chars/token), the same estimator the compiler enf
| agents | 567 | 40 | lazy |
| zed | 567 | 40 | lazy |
| copilot | 567 | 40 | lazy |
| cline | 539 | 539 | eager |
| cline | 567 | 40 | lazy |
| windsurf | 569 | 40 | lazy |
| gemini | 567 | 40 | lazy |
| aider | 560 | 560 | eager |
| agentsmd | 560 | 560 | eager |

**Standing tax:** ~40 tokens on a lazy target vs ~539 on an eager one — about **13× per session** for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.
**Standing tax:** ~40 tokens on a lazy target vs ~560 on an eager one — about **14× per session** for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.

## `review-checklist` — bare / unmanifested (no budget)

Expand All @@ -37,13 +37,13 @@ Token counts are estimates (~4 chars/token), the same estimator the compiler enf
| agents | 880 | 19 | lazy |
| zed | 880 | 19 | lazy |
| copilot | 880 | 19 | lazy |
| cline | 859 | 859 | eager |
| cline | 880 | 19 | lazy |
| windsurf | 881 | 19 | lazy |
| gemini | 880 | 19 | lazy |
| aider | 885 | 885 | eager |
| agentsmd | 885 | 885 | eager |

**Standing tax:** ~19 tokens on a lazy target vs ~859 on an eager one — about **45× per session** for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.
**Standing tax:** ~19 tokens on a lazy target vs ~885 on an eager one — about **47× per session** for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.

## Why this is the pitch

Expand Down
4 changes: 2 additions & 2 deletions packages/cli/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/cli/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "kitbash",
"version": "0.12.0",
"version": "0.13.0",
"description": "The package manager and compiler for AI agent skills — write once, run in every coding agent",
"license": "Apache-2.0",
"author": "Harsh Singh",
Expand Down
9 changes: 4 additions & 5 deletions packages/cli/scripts/benchmark.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,17 @@ function measure(tmp, target, skillName) {
return estimateTokens(read(`.claude/skills/${skillName}/SKILL.md`));
case "cursor":
return estimateTokens(read(`.cursor/rules/${skillName}.mdc`));
// zed reads the same vendor-neutral path and emits the same bytes, so it
// measures identically — listed rather than folded in, because a reader
// looking up "what does Zed cost" must find a row.
// zed and cline read the same vendor-neutral path and emit the same bytes,
// so they measure identically — listed rather than folded in, because a
// reader looking up "what does Zed cost" must find a row.
case "agents":
case "zed":
case "cline":
return estimateTokens(read(`.agents/skills/${skillName}/SKILL.md`));
case "copilot":
return estimateTokens(read(`.github/skills/${skillName}/SKILL.md`));
case "gemini":
return estimateTokens(read(`.gemini/skills/${skillName}/SKILL.md`));
case "cline":
return estimateTokens(read(`.clinerules/${skillName}.md`));
case "windsurf":
return estimateTokens(read(`.windsurf/rules/${skillName}.md`));
case "aider":
Expand Down
66 changes: 62 additions & 4 deletions packages/cli/scripts/test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ try {
check("agentsmd output exists", existsSync(agents));
check("copilot output exists", existsSync(join(tmp, ".github/skills/prereview/SKILL.md")));
check("copilot uses the lazy skills dir, not always-on instructions", !existsSync(join(tmp, ".github/instructions/prereview.instructions.md")));
check("cline output exists", existsSync(join(tmp, ".clinerules/prereview.md")));
check("cline compiles to the lazy skills path, not a .clinerules rule", existsSync(join(tmp, ".agents/skills/prereview/SKILL.md")));
check("cline no longer emits an always-on .clinerules rule", !existsSync(join(tmp, ".clinerules/prereview.md")));
check("windsurf output exists", existsSync(join(tmp, ".windsurf/rules/prereview.md")));
const windsurfOut = readFileSync(join(tmp, ".windsurf/rules/prereview.md"), "utf8");
check("windsurf rule is model_decision (lazy), not always-on", windsurfOut.startsWith("---\ntrigger: model_decision\n"), windsurfOut.slice(0, 120));
Expand All @@ -93,9 +94,9 @@ try {
check("agentsmd markers present", agentsContent.includes("<!-- kitbash:begin prereview -->") && agentsContent.includes("<!-- kitbash:end prereview -->"));
check("eager-load warning surfaced", compile.out.includes("cannot lazy-load"), compile.out);
// every eager target must report the standing cost; lazy targets must not
const eagerWarned = ["cline", "aider", "agentsmd"].every((t) => compile.out.includes(`→ ${t}: ${t} is eager and cannot lazy-load`));
const eagerWarned = ["aider", "agentsmd"].every((t) => compile.out.includes(`→ ${t}: ${t} is eager and cannot lazy-load`));
check("all eager targets report standing cost", eagerWarned, compile.out);
const lazyQuiet = ["claude-code", "cursor", "windsurf", "agents", "copilot", "gemini"].every((t) => !compile.out.includes(`→ ${t}: ${t} is eager`));
const lazyQuiet = ["claude-code", "cursor", "windsurf", "agents", "zed", "cline", "copilot", "gemini"].every((t) => !compile.out.includes(`→ ${t}: ${t} is eager`));
check("lazy targets do not warn", lazyQuiet, compile.out);

const recompile = run(["compile"], tmp);
Expand Down Expand Up @@ -253,7 +254,7 @@ try {
check("compile prunes stale claude output", !existsSync(join(tmp, ".claude/skills/prereview")), pruneCompile.out);
check("compile prunes stale cursor output", !existsSync(join(tmp, ".cursor/rules/prereview.mdc")), pruneCompile.out);
check("compile prunes stale command shim", !existsSync(join(tmp, ".claude/commands/prereview.md")), pruneCompile.out);
check("compile prunes stale cline output", !existsSync(join(tmp, ".clinerules/prereview.md")), pruneCompile.out);
check("compile prunes stale cline skill", !existsSync(join(tmp, ".agents/skills/prereview/SKILL.md")), pruneCompile.out);
const prunedAgents = readFileSync(join(tmp, "AGENTS.md"), "utf8");
check("compile prunes stale AGENTS.md section", !prunedAgents.includes("kitbash:begin prereview"), prunedAgents.slice(0, 200));
check("surviving skill section intact", prunedAgents.includes("kitbash:begin tidy-commits"));
Expand Down Expand Up @@ -746,6 +747,63 @@ try {
rmSync(zed, { recursive: true, force: true });
}

// --- cline: lazy skills, not an always-on rule, and never emitted twice ---

const clineFix = mkdtempSync(join(tmpdir(), "kitbash-cline-"));
try {
mkdirSync(join(clineFix, ".clinerules"));
// The output the pre-0.13.0 adapter left behind, plus a user's own rule file
// that must survive: pruning is scoped to kitbash's own generated output.
writeFileSync(
join(clineFix, ".clinerules/prereview.md"),
"<!-- generated by kitbash \u2014 do not edit; source: .kitbash/skills/prereview @ 0.1.0 -->\n\nold body\n",
);
writeFileSync(join(clineFix, ".clinerules/house-style.md"), "# My own rule\n\nKeep it.\n");

run(["init"], clineFix);
run(["install", `file:${fixture}`, "--yes"], clineFix);
const c = run(["compile"], clineFix);

check("cline: emits a lazy skill", existsSync(join(clineFix, ".agents/skills/prereview/SKILL.md")), c.out);
check("cline: upgrade prunes the stale always-on rule", !existsSync(join(clineFix, ".clinerules/prereview.md")), c.out);
check("cline: a user's own .clinerules file is untouched", existsSync(join(clineFix, ".clinerules/house-style.md")), c.out);
check("cline: no longer reports a standing token cost", !c.out.includes("→ cline: cline is eager"), c.out);

// The headline bug: .clinerules + .agents both present used to emit the same
// body twice, and Cline scans both paths — so the always-on copy defeated the
// lazy one. Sharing the path makes the duplicate structurally impossible.
mkdirSync(join(clineFix, ".agents"), { recursive: true });
const both = run(["compile"], clineFix);
const emitted = (both.out.match(/→ \.agents\/skills\/prereview\/SKILL\.md/g) ?? []).length;
check("cline + agents emit the shared skill once, not twice", emitted === 1 && !both.out.includes("conflict:"), both.out);
check("cline + agents leave no second copy anywhere", !existsSync(join(clineFix, ".cline/skills/prereview/SKILL.md")), both.out);

// The mirror-image conformance risk: Cline loads every skill on demand, so an
// eager-authored skill does not get what it asked for. That must be visible.
const eagerSrc = join(clineFix, "eager-src");
mkdirSync(eagerSrc);
writeFileSync(
join(eagerSrc, "skill.toml"),
'[skill]\nname = "always-on"\nversion = "0.1.0"\ndescription = "A rule that must always be resident"\n[context]\nbudget = 500\ndisclosure = "eager"\n',
);
writeFileSync(join(eagerSrc, "SKILL.md"), "Body of always-on.\n\nMore body.\n");
run(["install", `file:${eagerSrc}`, "--yes"], clineFix);
const eagerOut = run(["compile"], clineFix);
check(
"cline: eager-authored skill warns that Cline loads on demand",
eagerOut.out.includes('always-on → cline: authored disclosure = "eager"'),
eagerOut.out,
);
check("cline: that warning fails --strict", run(["compile", "--strict"], clineFix).status === 1);
check(
"cline: a lazy-authored skill draws no such warning",
!eagerOut.out.includes('prereview → cline: authored disclosure'),
eagerOut.out,
);
} finally {
rmSync(clineFix, { recursive: true, force: true });
}

// --- trust & review: pre-install review, --yes, [policy] allowlists ---

const trust = mkdtempSync(join(tmpdir(), "kitbash-trust-"));
Expand Down
52 changes: 45 additions & 7 deletions packages/cli/src/adapters.ts
Original file line number Diff line number Diff line change
Expand Up @@ -278,16 +278,54 @@ function zedConstraints(skill: LoadedSkill): string[] {
*/
const zed = skillDirAdapter("zed", ".agents/skills", (root) => existsSync(join(root, ".zed")), zedConstraints);

// Cline rule files are always loaded — eager.
const cline = fileAdapter(
/**
* Cline reads Agent Skills, and has since it grew a skills loader
* (`apps/vscode/src/core/storage/skill-directories.ts`): `.cline/skills`,
* `.clinerules/skills`, `.claude/skills` and `.agents/skills` are all scanned.
* Loading is progressive — name and description at startup, the body only when
* `use_skill` fires.
*
* Until 0.13.0 this adapter wrote `.clinerules/<name>.md`, which is a *rule*:
* always active, whole body in context, every session. That was wrong twice
* over. It billed Cline ~539 standing tokens for a skill Cline was willing to
* lazy-load for ~40, and in a repo that also had `.agents/` it emitted the same
* body to both paths — so Cline loaded the skill twice, and the always-on copy
* defeated the lazy one. A compiler whose entire claim is "cheapest loading
* mode the target actually supports" was picking the most expensive mode
* available and charging for it.
*
* Of the four scanned directories this emits to `.agents/skills`, the same
* vendor-neutral path `agents` and `zed` use, rather than the `.cline/skills`
* their docs nominate. Cline scans both equally, and sharing the path is what
* makes the duplicate structurally impossible: one file serves Cline, Codex,
* Cursor, Copilot and Zed at once, and a repo with several of their marker
* directories still compiles that skill exactly once.
*/
const cline = skillDirAdapter(
"cline",
[],
"eager",
(root) => existsSync(join(root, ".clinerules")),
(name) => `.clinerules/${name}.md`,
() => "",
".agents/skills",
(root) => existsSync(join(root, ".clinerules")) || existsSync(join(root, ".cline")),
clineConstraints,
);

/**
* The mirror of the bug above: Cline loads every skill on demand, so a skill
* authored `disclosure = "eager"` does not get the always-resident body its
* author asked for. Compiling that silently would trade one conformance
* violation (spec §2) for its reflection, so say it out loud.
*
* `.clinerules/<name>.md` would still deliver eager loading, and a
* disclosure-aware two-path adapter is the obvious extension — but nobody has
* asked for it, and a warning is the honest minimum in the meantime.
*/
function clineConstraints(skill: LoadedSkill): string[] {
if (skill.manifest.context.disclosure !== "eager") return [];
const { name } = skill.manifest.skill;
return [
`${name} → cline: authored disclosure = "eager", but Cline loads skills on demand — the body stays out of context until the model invokes it`,
];
}

/**
* Windsurf became Devin Desktop on 2026-06-02; `.devin/rules/` is the preferred
* path and `.windsurf/rules/` remains a supported fallback, so emit to whichever
Expand Down
Loading