Proposal: AgenticReplayBackend — replay mined tasks with a real agentic CLI in a throwaway worktree, gate on the diff

[Alphaxalchemy]

Motivation

Replay currently scores a single text completion: the candidate skill text + memory + task description go in as one prompt, one block of text comes out, and it is checked programmatically. (The only tool access is the small sanctioned attempt_with_tools loop for tool_called checks.)

That is honest for text-expressible behaviors — formatting, structure, phrasing, conventions — but it structurally cannot evaluate agentic behaviors: "run the tests, read the failure, fix the config" needs an environment that reacts. Much of what users actually want improved in coding agents lives in that multi-step space, and it also connects to #154: agentic tasks are exactly the ones whose success can't be expressed as a regex.

Proposal: AgenticReplayBackend

A replay backend that drives a real open agentic CLI, following the existing backend/attempt pattern:

1. Give the CLI the mined task in a throwaway git worktree/branch (sandboxed, disposable).
2. Let it actually run tools and edit files.
3. Gate on the diff rather than on output text: does it build? do tests pass? does the diff resemble the change the user eventually accepted in the original session?
4. Surface the diff in the report/dashboard for human review.

This gives real ground truth with no simulator-fidelity questions. (The heavier alternative — a language world model like Qwen-AgentWorld simulating environment feedback — is interesting for offline multi-step replay but adds a large unvalidated link to the evidence chain; probably a research direction rather than a v-next feature.)

Cost alignment

An obvious objection: SkillOpt's premise is cheap, token-efficient improvement of frozen agents, and a real agentic replay is orders of magnitude heavier than a single text completion (full CLI session, tool calls, builds, test runs). To be clear, this proposal is not "make every replay agentic." It's a tiered gate:

1. Cheap text replay stays the default for everything text-expressible — no change for the vast majority of candidates.
2. Agentic replay is invoked only for the candidate-rule category that cheap replay structurally cannot validate (the agentic behaviors from Mining is limited to programmatic success checks — intent-level improvements get dropped or Goodharted into shallow proxy rules #154), and only for candidates that have already survived the cheap filters.
3. It runs under an explicit per-night budget (e.g., at most N agentic replays per night, token-capped), opt-in per user.

Two further points on alignment with the original premise:

• SkillOpt's cheapness claim is fundamentally about the deployed artifact: a small text file improving a frozen model at zero inference-time cost. This proposal doesn't touch that — the skill that comes out is exactly as cheap to deploy. The added cost is one-time validation expense, amortized over every future session, which is consistent with the existing design (the optimizer itself already uses a frontier model).
• The pitch is to spend expensive validation only where cheap validation is worthless. A wrong-but-accepted agentic rule costs more — in degraded future sessions — than the validation would have. An unvalidated rule that passes a regex gate but breaks real workflows is the truly expensive outcome.

Bigger picture

This is one of the key steps toward SkillOpt functioning as on-the-job training for a specific job role: learning from real work, evaluated on real work.

Happy to help spec this out if there's maintainer interest.

[Yif-Yang]

Thanks for the thoughtful proposal. We agree that there is a real gap between text-only replay and the multi-step behaviors users often want coding agents to improve. A tiered design — keeping cheap text replay as the default and invoking agentic replay only for candidates that require an interactive environment — is a sensible direction to explore.

The most important boundary is that a throwaway Git worktree is disposable version-control state, but it is not a security sandbox. A real agentic replay backend would still need isolation from the host, user credentials and configuration, unrelated files, network access, and shared Git metadata. It would also need hard limits on turns, time, processes, concurrency, and provider spend, with reliable cleanup after every outcome.

We would also treat task verification as the primary gate rather than diff similarity. Builds, tests, task-specific verifiers, allowed-file boundaries, and regression checks provide stronger evidence of correctness. Similarity to the change accepted in the original session can be useful context for a judge or human reviewer, but it should remain secondary because multiple substantially different diffs may solve the same task correctly.

We are interested in exploring this direction and would welcome a community design or draft PR. A safe first slice could be:

• fully opt-in and limited to one agent backend and one narrow task category;
• a pinned starting commit and reproducible fixture/environment;
• execution inside a real container or OS sandbox with isolated home/configuration and network disabled by default;
• deterministic tests or a task verifier as the main acceptance signal;
• explicit token, turn, time, and replay-count budgets; and
• a staged diff/report for review, without automatic adoption.

Starting with a small end-to-end prototype would let us evaluate the evidence quality, cost, and safety model before generalizing it to arbitrary mined tasks. We would be happy to work with you or other interested contributors on refining that scope.

[siddhivmd]

Hi, I'd like to help implement this. I'm newer to the codebase — could I take a first pass at a small piece, like the sandboxed worktree setup (step 1)? Happy to follow whatever pattern you'd prefer.