Problem — conversation content is written to logs today
JudgeClassifier::verdict logs every unavailable verdict with the error's raw Display:
tracing::warn!(target: "libsy", judge_model, error = %error, "judge verdict unavailable; ...");
That is not safe here, and the leak is concrete rather than theoretical:
LibsyError::ClientCall renders as "client call to target {target:?} failed: {source}";
- its source
LlmClientError::UpstreamHttp renders as "upstream returned HTTP {status}: {body}".
body is the upstream error body verbatim. Providers routinely quote the offending request back
in it, so any 4xx from a judge target writes conversation text into the operator's logs. Boxed
transport, decode, and FFI sources can carry the same.
This runs on a per-turn path, in the component most likely to be pointed at a self-hosted endpoint
with unfamiliar error formats.
Proposal
A robustness module with safe_error_summary / safe_client_error, used at the judge's warn site.
Each case contributes only vetted operational detail — the failure's class, the target or model it
concerns, and a status code where one exists:
| Error |
Logged as |
UpstreamHttp { status, body } |
upstream HTTP 400 — body dropped |
ContextWindowExceeded { model, message } |
context window exceeded for model weak — message dropped |
Transport / InvalidResponse / Timeout / Ffi |
class only; boxed source dropped |
External { operation, source } |
{operation} failed — static label kept, source dropped |
AlgorithmError { message } |
verbatim — libsy builds it from static text |
libsy's own AlgorithmError is reported in full deliberately: it is built from a static string plus a
serde position and is the one message an operator needs to diagnose a bad judge reply. This mirrors
the safe_types idea from the Python implementation this is ported from.
Design note — the exhaustive match earned its keep
The match over LibsyError is deliberately exhaustive rather than _ => error.to_string(), so a new
variant cannot start leaking by omission. That paid off immediately: the compiler caught
External { operation, source }, which pairs a safe static label with an unvetted boxed source from a
user extension, and which a catch-all would have silently rendered in full.
Test approach
Each test asserts both that the summary omits the sensitive text and that the error's
Display still contains it — so the tests fail if the redaction is ever bypassed, not merely if it
changes shape.
Scope
3 files, +177/−8. New crates/libsy/src/algorithms/util/robustness.rs (with 6 tests), its module
declaration, and 20 lines at the judge's log site. No public API change, no config surface.
Why this is its own PR
Independent of the other two classifier changes: it is a data-hygiene defect on the logging path,
with no overlap in code path or surface. It is also the most clearly severity-driven of the three,
so it should not be queued behind a prompt-tuning discussion or a breaking-config debate.
Validation
| Gate |
Result |
cargo test --workspace |
646 passed (baseline 640 + 6 new) |
cargo clippy --workspace --all-targets -- -D warnings |
clean |
cargo fmt --check |
clean |
uv run pytest tests/ |
unchanged from baseline |
Follow-up worth raising separately
Only the judge path is fixed here. Other %error log sites in the workspace may have the same
exposure and are worth an audit — deliberately out of scope so this stays reviewable.
Problem — conversation content is written to logs today
JudgeClassifier::verdictlogs every unavailable verdict with the error's rawDisplay:That is not safe here, and the leak is concrete rather than theoretical:
LibsyError::ClientCallrenders as"client call to target {target:?} failed: {source}";LlmClientError::UpstreamHttprenders as"upstream returned HTTP {status}: {body}".bodyis the upstream error body verbatim. Providers routinely quote the offending request backin it, so any 4xx from a judge target writes conversation text into the operator's logs. Boxed
transport, decode, and FFI sources can carry the same.
This runs on a per-turn path, in the component most likely to be pointed at a self-hosted endpoint
with unfamiliar error formats.
Proposal
A
robustnessmodule withsafe_error_summary/safe_client_error, used at the judge's warn site.Each case contributes only vetted operational detail — the failure's class, the target or model it
concerns, and a status code where one exists:
UpstreamHttp { status, body }upstream HTTP 400— body droppedContextWindowExceeded { model, message }context window exceeded for model weak— message droppedTransport/InvalidResponse/Timeout/FfiExternal { operation, source }{operation} failed— static label kept, source droppedAlgorithmError { message }libsy's own
AlgorithmErroris reported in full deliberately: it is built from a static string plus aserde position and is the one message an operator needs to diagnose a bad judge reply. This mirrors
the
safe_typesidea from the Python implementation this is ported from.Design note — the exhaustive match earned its keep
The match over
LibsyErroris deliberately exhaustive rather than_ => error.to_string(), so a newvariant cannot start leaking by omission. That paid off immediately: the compiler caught
External { operation, source }, which pairs a safe static label with an unvetted boxed source from auser extension, and which a catch-all would have silently rendered in full.
Test approach
Each test asserts both that the summary omits the sensitive text and that the error's
Displaystill contains it — so the tests fail if the redaction is ever bypassed, not merely if itchanges shape.
Scope
3 files, +177/−8. New
crates/libsy/src/algorithms/util/robustness.rs(with 6 tests), its moduledeclaration, and 20 lines at the judge's log site. No public API change, no config surface.
Why this is its own PR
Independent of the other two classifier changes: it is a data-hygiene defect on the logging path,
with no overlap in code path or surface. It is also the most clearly severity-driven of the three,
so it should not be queued behind a prompt-tuning discussion or a breaking-config debate.
Validation
cargo test --workspacecargo clippy --workspace --all-targets -- -D warningscargo fmt --checkuv run pytest tests/Follow-up worth raising separately
Only the judge path is fixed here. Other
%errorlog sites in the workspace may have the sameexposure and are worth an audit — deliberately out of scope so this stays reviewable.