Fail loud on unparseable triage output instead of dropping the batch#121
Open
thejesh23 wants to merge 1 commit into
Open
Fail loud on unparseable triage output instead of dropping the batch#121thejesh23 wants to merge 1 commit into
thejesh23 wants to merge 1 commit into
Conversation
triageBatch parsed the model's verdict array with a bare `JSON.parse`
wrapped in `try { … } catch {}`, leaving `verdicts = []` on any error.
A common trailing-comma or truncated array was therefore swallowed: the
loop ran zero times, every finding in the (up to 30-finding) batch stayed
un-triaged, and the batch was still reported as a success ("N triaged")
— indistinguishable from a clean "nothing to do" run.
Route triage through the same tolerant parse + fail-loud path the
investigate/revalidate agents already use: `parseAgentJsonArray`
(jsonrepair-backed) recovers minor malformations, and genuinely
unparseable output is dumped via `writeParseFailureDebug` and rethrown so
the outer catch marks the batch failed rather than silently empty.
`parseAgentJsonArray` is exported from agents/shared.ts for reuse.
Adds regression tests: a trailing-comma array is recovered and triaged; a
non-array reply surfaces a failed batch instead of "0 triaged".
|
@thejesh23 is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
triageBatchnow parses the model's verdict array through the same tolerant, fail-loud path the investigate/revalidate agents already use —parseAgentJsonArray(jsonrepair-backed) — instead of a bareJSON.parsein an empty catch. Genuinely unparseable output is dumped viawriteParseFailureDebugand rethrown so the outer catch marks the batch failed.parseAgentJsonArrayis exported fromagents/shared.tsfor reuse.Why
The old code left
verdicts = []on any parse error, so a routine trailing-comma or truncated array silently lost the triage for the entire (up to 30-finding) batch while still printingBatch N/M: 0 triagedas success — indistinguishable from a clean "nothing to triage" run.agents/shared.tsalready documents this exact failure mode for the investigate path ("a malformed response is otherwise indistinguishable from a clean 'found nothing' run") and guards against it; triage now matches that behavior.Fixes #120
Verification
pnpm testpasses (added regression tests: a trailing-comma array is recovered and triaged; a non-array reply surfaces a failed batch instead of "0 triaged". Both fail onmainand pass with this change)pnpm lintpassespnpm knippassesAlso ran
pnpm -r build(typecheck) andpnpm test:bundle— both green.Notes for reviewer
Minor malformations that
jsonrepaircan recover are now triaged rather than dropped; only genuinely unrecoverable output fails the batch (and writes aparse-error-revalidate-*.txtdebug dump, consistent with the other agents). No change to the verdict-application logic itself.