Mining is limited to programmatic success checks — intent-level improvements get dropped or Goodharted into shallow proxy rules

[Alphaxalchemy]

ORIGIN & MOTIVATION

I'm exploring using SkillOpt as a step toward "on-the-job training" for AI Employees learning a specific job role from real work sessions. This issue is about what looks like a structural ceiling on that goal in the mining stage. (Related but distinct from #67, which covers reward-hacking on the gate side; this is about the miner side.)

Problem

The task miner only retains a candidate task if success can be expressed as a programmatic check (regex-style assertions). That means behaviors like "understand my intent better" don't compress into a check, so they are either:

1. dropped the most valuable improvement signal never enters the pipeline, or
2. mangled into a shallow proxy a phrasing/format preference observed in one session gets distilled into a hyper-literal rule that is then robotically applied to all future outputs. Goodhart's law in miniature: the check becomes the target.

I've observed the second failure mode in my own runs: a one-off summary phrasing preference became a rigid rule stamped onto every future summary.

If mining is restricted to regex-expressible outcomes, the system can only ever improve the regex-expressible slice of agent behavior, which is a small subset of what users actually repeat-and-rephrase about in real sessions.

Possible directions (heuristics, not designs)

1. Divergence-point detection: mine topic-shift structure in transcripts (user descends into a rabbit hole on a sub-issue → resolves it → conversation returns to the main thread) as natural task boundaries and implicit failure signals, instead of requiring a programmatic outcome check.
2. Clean-context replay comparison: a second instance with fresh context attempts the reconstructed task; a judge compares its output against what the user ultimately accepted in the original transcript, rather than against a regex proxy.
3. Baseline-instance divergence: an instance seeded only with the user's global rules/skills replays the session's user prompts in order until its output clearly and definitely diverges from the transcript — the divergence point marks where a learnable behavior lives, and bounds the task to mine.

Question for maintainers

Is there interest in supporting judge-based (non-programmatic) success checks in mining/replay, and what evidence standard would you consider acceptable for gating on them?

[Yif-Yang]

Thanks for describing this failure mode so concretely. A one-off preference being over-generalized into a permanent rule is a real concern, especially when SkillOpt-Sleep is used with transcripts from day-to-day work.

One implementation detail may help align the discussion: the current miner is not limited to tasks that can be expressed as regexes or other programmatic assertions. It prefers locally verifiable rule checks when appropriate, but if no suitable programmatic check exists and it can produce a meaningful free-text rubric, the task is retained with reference_kind="rubric". During replay, a real backend uses an LLM judge to score that rubric from 0 to 1, with 0.8 or above treated as a hard pass. A task is dropped only when neither usable checks nor a rubric can be formed. So judge-based, non-programmatic evaluation already exists in the current pipeline.

That said, this does not eliminate the problem you observed. The miner currently infers a reusable intent and rubric in one pass from a compressed session digest. It has neither divergence-point detection nor clean-context replay comparison, and it cannot reliably distinguish a temporary wording preference from a stable preference. It can therefore still produce a shallow proxy that the optimizer learns to satisfy rather than the underlying intent.

Could you share a sanitized minimal reproduction? The most useful artifacts would be:

• the relevant user correction, assistant final answer, and feedback signals;
• the mined TaskRecord, especially intent, reference_kind, and reference/judge;
• the rule that was ultimately written into the skill or memory;
• the backend, model, and relevant configuration.

That would help us determine whether the failure originates in task-boundary detection, rubric construction, replay/judging, or consolidation.

We are interested in improving this area, and we welcome community PRs that explore these ideas. We would be happy to work with contributors on the scope and evaluate an incremental design together. Clean-context replay comparison in particular seems like a promising experiment.

For eventual integration of a new judge mechanism, we would want frozen rubrics with provenance, held-out tasks or sessions, calibration against a small human-labeled set, and an abstain/fail-closed path for ambiguous cases. Programmatic checks should remain preferred where they genuinely fit, with semantic judges complementing rather than unconditionally replacing them. These are longer-term acceptance goals, not requirements for sharing an initial reproduction or opening an early draft PR.

We would prefer to start with a reproducible case and a small, testable change rather than introduce an entirely new pipeline at once.