Skip to content

[unit: providers] Research document#305

Open
jayfalls wants to merge 1 commit intomainfrom
providers-design-1
Open

[unit: providers] Research document#305
jayfalls wants to merge 1 commit intomainfrom
providers-design-1

Conversation

@jayfalls
Copy link
Copy Markdown
Owner

@jayfalls jayfalls commented May 4, 2026

Summary

Full research document for the Providers unit covering 7 open architectural questions:

  1. Routing strategies - Round-Robin, Sequential, Failover (with circuit-breaker). Deferred: weighted, latency-based, cost-optimized.
  2. Streaming - Internal-only in this unit. Gateway uses streaming provider APIs but buffers full responses for NATS consumers.
  3. Retry/Timeout - Exponential backoff with full jitter, base 1s, cap 8s, 2 retries. Per-provider configurable timeouts (60s/120s/300s defaults).
  4. Token counting - Provider-reported only, normalised to inclusive model at adapter layer.
  5. NATS payload schema - Versioned JSON Go structs (LLMRequest / LLMResponse) with normalised error codes.
  6. Encryption - Envelope encryption per-provider DEK wrapped by master KEK for zero-downtime key rotation.
  7. Model feature detection - Static capability catalog + dynamic model list fetch + manual override.

Delivered: design/units/providers/research.md

Covers routing strategies (Round-Robin, Sequential, Failover with circuit-breaker),
internal-only streaming, retry/timeout strategy, token counting (provider-reported),
NATS payload schema, envelope encryption, and model feature detection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant