Skip to content

fix(run): raise review findings JSON cap 5000→20000 chars#167

Open
johnkattenhorn wants to merge 1 commit into
LarsCowe:mainfrom
johnkattenhorn:pr-review-findings-limit
Open

fix(run): raise review findings JSON cap 5000→20000 chars#167
johnkattenhorn wants to merge 1 commit into
LarsCowe:mainfrom
johnkattenhorn:pr-review-findings-limit

Conversation

@johnkattenhorn
Copy link
Copy Markdown

Problem

In run_review_loop (ralph/ralph_loop.sh:1957, 1963) the extracted JSON between the ---REVIEW_FINDINGS--- and ---END_REVIEW_FINDINGS--- markers is truncated to 5000 chars before being passed to jq for validation.

findings_json=$(sed -n '/---REVIEW_FINDINGS---/,/---END_REVIEW_FINDINGS---/{//!p;}' "$review_output_file" 2>/dev/null | tr -d '\n' | head -c 5000)

Any review that produces more than 5000 chars of JSON — which is trivial to hit with a project-specific REVIEW_PROMPT.md that asks for file, line, category, issue, and a substantive suggestion per detail entry — gets cut mid-object. jq then rejects it as malformed syntax and the whole batch of findings is silently dropped:

[2026-04-17 12:18:21] [WARN] Review findings JSON is malformed — skipping

Ralph never sees the findings. The review loop burns full tokens but provides no signal to the next implementation loop.

Repro

Observed in a real project running REVIEW_MODE=ultimate with a customised REVIEW_PROMPT.md. Six reviews ran:

Review JSON size Outcome
Loop 2 2991 chars ✅ parsed
Loop 4 5500 chars ❌ malformed (truncated at 5000)
Loop 5 5931 chars ❌ malformed
Loop 6 5546 chars ❌ malformed
Loop 7 5073 chars ❌ malformed (just over)
Loop 8 4162 chars ✅ parsed

Every review over 5000 chars failed. The lost findings included real HIGH-severity items the reviewer caught (missing cross-user isolation tests, pattern drift, architectural deviations) that Ralph would otherwise have addressed in the following loop.

Fix

Raise both head -c 5000 occurrences to head -c 20000. That's comfortable headroom for ~8–12 detailed findings per review while preserving the cap as a safety rail against a runaway model response.

No behaviour change for smaller reviews.

Test plan

  • Manual: re-ran a previously-failing review_loop output file through the parser with the new limit — JSON now parses cleanly, jq .issues_found returns the correct count.
  • No existing tests hit the truncation path (all test fixtures are small), so no test updates needed.
  • Would welcome a suggestion on where a bats test guarding this would belong — tests/bash/ structure is mature but I didn't want to speculate.

Related

Discovered alongside #165 (prepare-script fix) while running a real project against a custom REVIEW_PROMPT.md. Both are independent.

The review loop parser truncates the extracted REVIEW_FINDINGS JSON
to 5000 chars before handing it to jq for validation. In practice any
review with 5+ detailed findings (each with a non-trivial `suggestion`
field) exceeds 5000, getting cut mid-object, and then failing jq's
syntax check.

The only user-visible signal is a WARN line in ralph.log:
  Review findings JSON is malformed — skipping

Findings are silently dropped, never reach the next implementation
loop, and Ralph never acts on them. Observed in a real project with
`REVIEW_MODE=ultimate` and a project-specific REVIEW_PROMPT.md: 4 of 6
reviews lost their output this way (loops producing 5073, 5546, 5931,
5500 chars of JSON — all just over the cap).

20000 chars (≈8–12 detailed findings) gives comfortable headroom for
richer review prompts while keeping the cap in place as a safety rail
against a runaway Claude response. No behavior change for smaller
reviews.

Also applies to the fallback extraction path (when the first sed does
not find the markers in the raw file and we extract via `jq -r .result`).
johnkattenhorn added a commit to johnkattenhorn/bmalph that referenced this pull request Apr 17, 2026
Patch release for the 5000-char REVIEW_FINDINGS truncation fix.
Upstream PR: LarsCowe#167
johnkattenhorn added a commit to johnkattenhorn/bmalph that referenced this pull request Apr 17, 2026
Patch release for the 5000-char REVIEW_FINDINGS truncation fix.
Upstream PR: LarsCowe#167
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant