Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Changed

- **Switchyard HTTP transport limits** — provider redirects are rejected,
connection and read-inactivity timeouts are enforced, buffered success and
error bodies are bounded, and oversized SSE events fail before unbounded line
buffering. Provider error bodies remain available in typed errors but are no
longer included in their default display text.

### Removed

- **Latency-aware router** — the `latency_service` route type and its
Expand Down
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions crates/libsy-llm-client/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ tokio.workspace = true
tracing.workspace = true

[dev-dependencies]
bytes = "1"
futures.workspace = true
http = "1"
http-body = "1"
wiremock = "0.6"
13 changes: 13 additions & 0 deletions crates/libsy-llm-client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ fn build_multi_format_client(
`authorization` / `x-api-key` / `anthropic-version` / `content-type`. So a
caller's placeholder credential never overrides the backend's real key.
- Per-backend static headers go in `HttpBackendConfig::extra_headers`.
Its `Debug` output includes header names but redacts all values.
- Per-target top-level request defaults go in `HttpBackendConfig::extra_body`.
The merge is shallow and fields already present in the request take precedence.
- `HttpBackendConfig::max_retries` controls additional attempts after retryable
Expand All @@ -183,6 +184,12 @@ fn build_multi_format_client(
Retries replay the same upstream request. A transport failure can therefore
duplicate a request that the provider processed but did not finish returning,
and the retry budget plus capped `Retry-After` delays determines total latency.
Configured provider requests do not follow redirects. Connections time out
after 10 seconds, and a 120-second read timeout resets after every successful
read so active streams may continue while stalled providers are released. Use
`TranslatingLlmClient::new_with_transport_config` with an
`HttpTransportConfig` to change either timeout; setting `read_timeout` to `None`
disables the idle-read timeout for providers that may remain silent for longer.

## Errors

Expand All @@ -202,6 +209,12 @@ and the retry budget plus capped `Retry-After` delays determines total latency.
| `InvalidResponse { source }` | the upstream response could not be decoded |
| `Other(source)` | a client-specific failure outside the shared categories |

Buffered success bodies are capped at 64 MiB and HTTP error bodies at 64 KiB.
Typed HTTP errors retain the bounded body for explicit handling, but their
default display text includes only the status so provider content is not copied
into ordinary logs or spans. The shared stream decoder rejects an SSE frame
larger than 8 MiB before its line buffer can grow without bound.

[`switchyard_protocol::Request`]: ../libsy-protocol
[`switchyard_protocol::Response`]: ../libsy-protocol
[`libsy-proxy`]: ../libsy-proxy
Expand Down
42 changes: 40 additions & 2 deletions crates/libsy-llm-client/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,27 @@ pub struct HttpBackendConfig {

impl fmt::Debug for HttpBackendConfig {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let extra_header_names = self.extra_headers.keys().collect::<Vec<_>>();
let base_url = redacted_base_url(&self.base_url);
f.debug_struct("HttpBackendConfig")
.field("base_url", &self.base_url)
.field("base_url", &base_url)
.field("api_key", &self.api_key.as_ref().map(|_| "[REDACTED]"))
.field("extra_headers", &self.extra_headers)
.field("extra_header_names", &extra_header_names)
.field("extra_body_keys", &self.extra_body.keys())
.field("max_retries", &self.max_retries)
.finish()
}
}

fn redacted_base_url(base_url: &str) -> String {
let Ok(mut url) = reqwest::Url::parse(base_url) else {
return "[INVALID URL]".into();
};
let _ = url.set_username("");
let _ = url.set_password(None);
url.into()
}

/// A configured upstream backend, one variant per built-in wire format.
///
/// The variant fixes the wire format, URL path, and auth scheme together so no
Expand Down Expand Up @@ -217,6 +228,21 @@ mod tests {
assert_eq!(backend.url(), "https://api.openai.com/v1/chat/completions");
}

#[test]
fn debug_redacts_base_url_userinfo() {
let debug = format!("{:?}", config("https://user:pass@provider.example/v1"));
assert!(debug.contains("provider.example/v1"));
assert!(!debug.contains("user"));
assert!(!debug.contains("pass"));
}

#[test]
fn debug_does_not_emit_an_invalid_base_url() {
let debug = format!("{:?}", config("not a valid url with a secret"));
assert!(debug.contains("[INVALID URL]"));
assert!(!debug.contains("secret"));
}

#[test]
fn openai_chat_url_tolerates_trailing_slash_and_existing_suffix() {
assert_eq!(
Expand Down Expand Up @@ -281,6 +307,18 @@ mod tests {
assert!(!Backend::OpenAiResponses(config("x")).is_anthropic());
}

#[test]
fn debug_redacts_static_header_values() {
let mut config = config("https://provider.example/v1");
config
.extra_headers
.insert("authorization".into(), "Bearer provider-secret".into());

let debug = format!("{config:?}");
assert!(debug.contains("authorization"));
assert!(!debug.contains("provider-secret"));
}

#[test]
fn wire_format_matches_variant() {
assert_eq!(
Expand Down
Loading
Loading