Skip to content

fix(libsy): redact upstream error content from judge warning logs #278

Description

@gburachas

Problem — conversation content is written to logs today

JudgeClassifier::verdict logs every unavailable verdict with the error's raw Display:

tracing::warn!(target: "libsy", judge_model, error = %error, "judge verdict unavailable; ...");

That is not safe here, and the leak is concrete rather than theoretical:

  • LibsyError::ClientCall renders as "client call to target {target:?} failed: {source}";
  • its source LlmClientError::UpstreamHttp renders as "upstream returned HTTP {status}: {body}".

body is the upstream error body verbatim. Providers routinely quote the offending request back
in it, so any 4xx from a judge target writes conversation text into the operator's logs. Boxed
transport, decode, and FFI sources can carry the same.

This runs on a per-turn path, in the component most likely to be pointed at a self-hosted endpoint
with unfamiliar error formats.

Proposal

A robustness module with safe_error_summary / safe_client_error, used at the judge's warn site.

Each case contributes only vetted operational detail — the failure's class, the target or model it
concerns, and a status code where one exists:

Error Logged as
UpstreamHttp { status, body } upstream HTTP 400 — body dropped
ContextWindowExceeded { model, message } context window exceeded for model weak — message dropped
Transport / InvalidResponse / Timeout / Ffi class only; boxed source dropped
External { operation, source } {operation} failed — static label kept, source dropped
AlgorithmError { message } verbatim — libsy builds it from static text

libsy's own AlgorithmError is reported in full deliberately: it is built from a static string plus a
serde position and is the one message an operator needs to diagnose a bad judge reply. This mirrors
the safe_types idea from the Python implementation this is ported from.

Design note — the exhaustive match earned its keep

The match over LibsyError is deliberately exhaustive rather than _ => error.to_string(), so a new
variant cannot start leaking by omission. That paid off immediately: the compiler caught
External { operation, source }, which pairs a safe static label with an unvetted boxed source from a
user extension, and which a catch-all would have silently rendered in full.

Test approach

Each test asserts both that the summary omits the sensitive text and that the error's
Display still contains it — so the tests fail if the redaction is ever bypassed, not merely if it
changes shape.

Scope

3 files, +177/−8. New crates/libsy/src/algorithms/util/robustness.rs (with 6 tests), its module
declaration, and 20 lines at the judge's log site. No public API change, no config surface.

Why this is its own PR

Independent of the other two classifier changes: it is a data-hygiene defect on the logging path,
with no overlap in code path or surface. It is also the most clearly severity-driven of the three,
so it should not be queued behind a prompt-tuning discussion or a breaking-config debate.

Validation

Gate Result
cargo test --workspace 646 passed (baseline 640 + 6 new)
cargo clippy --workspace --all-targets -- -D warnings clean
cargo fmt --check clean
uv run pytest tests/ unchanged from baseline

Follow-up worth raising separately

Only the judge path is fixed here. Other %error log sites in the workspace may have the same
exposure and are worth an audit — deliberately out of scope so this stays reviewable.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions