Skip to content

fix(libsy): put judge system prompt in instructions, not messages - #312

Merged
linj-glitch merged 2 commits into
mainfrom
fix/libsy-structured-judge-system-instruction
Aug 5, 2026
Merged

fix(libsy): put judge system prompt in instructions, not messages#312
linj-glitch merged 2 commits into
mainfrom
fix/libsy-structured-judge-system-instruction

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

PR #204 refactored CapabilityJudge to a StructuredJudge type alias, but StructuredJudge::build_request inserts the system prompt into messages as a Role::System entry instead of LlmRequest.instructions. This silently reverted the fix from PR #195 and removed its test.

For OpenAI-compatible backends the wire output happens to be identical, but for Anthropic backends a Role::System message in messages serializes as a user turn — the system field ends up empty. Fix StructuredJudge::build_request to use instructions (as InstructionBlock), and update the tests in llm_class.rs and escalation.rs that were asserting the old messages-based shape.

@linj-glitch
linj-glitch requested a review from a team as a code owner August 5, 2026 19:20
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

🚀 View preview at
https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-312/

Built to branch gh-pages at 2026-08-05 19:41 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The structured judge request now sends contract prompts through system instruction blocks. Conversation messages remain separate. Mock clients and request-shape tests now validate the updated protocol structure.

Changes

Structured judge request migration

Layer / File(s) Summary
Build requests with instruction blocks
crates/libsy/src/algorithms/util/llm_judge.rs
StructuredJudge::build_request places the contract prompt in a system InstructionBlock. Classifier-generated messages remain in messages.
Update structured request assertions
crates/libsy/src/algorithms/llm_class.rs, crates/libsy/src/algorithms/util/escalation.rs
Tests extract prompts from instruction blocks and assert separate system instructions and conversation messages.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A rabbit sees prompts hop to their place,
System instructions lead the race.
Messages follow in tidy rows,
Tests now check the path it shows.
Hop, hop—structured requests compose!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: moving the judge system prompt from messages to instructions.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

…ing test

Signed-off-by: Lin Jia <linj@nvidia.com>

@ryan-lempka ryan-lempka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@linj-glitch
linj-glitch merged commit f4aa288 into main Aug 5, 2026
19 checks passed
@linj-glitch
linj-glitch deleted the fix/libsy-structured-judge-system-instruction branch August 5, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants