RFC: Kai prompt & instruction governance - #618
Draft
claude[bot] wants to merge 3 commits into
Draft
Conversation
|
|
||
| Kai's behavior is shaped by instructions authored across **three repos with no contract between them**, and the same guidance is duplicated across them where it then drifts. A 2026-07-09 audit of every prompt surface found **19 verified conflicts** (5 high / 8 medium / 6 low). | ||
|
|
||
| The three layers that compose Kai's live instructions: |
Contributor
There was a problem hiding this comment.
I like the clear definition of the 3 layers here.
|
|
||
| ## Resolution Strategy | ||
|
|
||
| ### 1. Ownership contract |
Contributor
There was a problem hiding this comment.
IMHO, another option here is to go the path of having a plugin companion to the MCP. It's what Slack & Linear do (among others). Ofc, we would then need to decide where the Keboola MCP plugin lives (either in this repo or in the AI Kit)
|
|
||
| ### 2. Runtime-scoped overlays | ||
|
|
||
| ai-kit skills gain frontmatter runtime targeting (e.g. `targets: [claude-code, kai-sandbox, mcp-only]`) and use explicit `In Kai:` / `In standalone CLI:` callouts for content that diverges. Base content = shared core; overlays = runtime specifics — the model is base + overlay (like kustomize overlays or the CSS cascade), not copy-and-diverge. The existing vendoring step (`packages/kai-agent-sandbox/scripts/vendor-external-skills.ts`) selects/strips sections by target at vendor time. This directly resolves the Streamlit-first-vs-never-Streamlit, CDN-vs-React, and cache-vs-live-query conflicts: the base stops asserting a universal default that Kai contradicts. |
Contributor
There was a problem hiding this comment.
I don't know how scalable this is. Seems annoying to maintain.
|
|
||
| Today the live prompt is assembled at runtime from four sources — `prompt-builder.ts` (base identity + skill gate + platform `llm_instruction` + per-request context), the vendored ai-kit skill markdown, the first-party `kai-extras` skill, and the MCP tool docstrings injected by the client — and **no human or test ever sees the whole assembled string**. Each fragment is reviewed in isolation, in a different repo, which is exactly how the high-severity contradictions slipped through. | ||
|
|
||
| Proposal: a generator + CI check that assembles and emits the composed prompt as a committed artifact, mirroring the pattern `mcp-server` already uses for tool docs (`python -m keboola_mcp_server.generate_tool_docs` → `TOOLS.md`, gated by `tox -e check-tools-docs`, which runs the generator then `git diff --exit-code TOOLS.md` so CI fails if it is stale). |
Contributor
There was a problem hiding this comment.
Sounds nice, but I'd need to understand more closely how it works.
…down, expand composed-prompt mechanism
…defer guards until drift recurs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Jordan Burger · Slack thread
Description
Linear: to be filed (follows the 2026-07-09 cross-repo prompt audit; related AI-3531, AI-3532)
Change Type
Summary
Proposes a governance model for the instructions that drive Kai, which today live across three repos (
keboola/ai-kitskills,keboola/mcp-servertool docstrings + project system prompt, andkeboola/ui's kai-agent) with no contract between them. A 2026-07-09 audit found 19 verified conflicts; the root cause is ai-kit skills authored for standalone Claude Code / kbagent being build-time vendored into Kai's different runtime (MCP + React sandbox).Before: a prompt fragment is reviewed in isolation in one repo; a stale cross-reference or a standalone default that Kai contradicts ships silently, and nobody sees the composed prompt Kai actually receives.
After: each instruction has one owner, runtime-specific content is an explicit overlay, the fully-composed prompt is emitted as a diffable CI artifact, every skill/tool/file reference is linted, and prompt-consistency behavior is gated by KaiBench evals.
Adds
feature_spec/kai_prompt_governance/RFC.mdfollowing the repo's RFC template. The RFC specifies: (1) an ownership contract (capability facts → mcp-server, domain how-to → ai-kit, runtime/persona → ui) + CODEOWNERS; (2) runtime-scoped skill overlays selected at vendor time; (3) a composed-prompt build artifact +git diff --exit-codeCI gate mirroringtox -e check-tools-docs/TOOLS.md; (4) a cross-reference linter split across mcp-server (tool refs) and ui (skill/file refs); (5) a first-class prompt-consistency track built on KaiBench's existingforbidden_tools/expected_tools/--repeat/regression machinery, with four framework gaps to close and a dependency on a Kai backend endpoint that exposes the composed prompt.Doc-only change (no code).
Testing
Streamable-HTTPtransports) — N/A, doc-only changeChecklist
Generated by Claude Code