feat(backend): add authentication support to OpenAIHTTPBackend #491
Conversation
Add api_key, bearer_token, and headers parameters to OpenAIHTTPBackend
to enable authentication with OpenAI-compatible servers that require it.
Changes:
- Add api_key, bearer_token, and headers params to __init__
- Read default values from settings.openai when not explicitly provided
- Apply auth headers to backend validation (/health endpoint)
- Apply auth headers to available_models() requests
- Merge default headers with request-specific headers in resolve()
This fixes issues when using GuideLLM with LiteLLM proxy or other
OpenAI-compatible servers that require authentication on all endpoints
including /health.
Usage:
# Via environment variables (recommended)
export GUIDELLM__OPENAI__API_KEY=sk-xxx
guidellm benchmark run --target http://litellm:4000 ...
# Via backend kwargs
guidellm benchmark run --target http://litellm:4000 \
--backend-kwargs '{"api_key": "sk-xxx"}' ...
…ire la taille d'image
sjmonson
left a comment
sjmonson
left a comment
There was a problem hiding this comment.
Thanks for this, I think it was accidentally wiped out as a part of v0.4.0. See necessary changes below.
| self.model = model | ||
|
|
||
| # Build default headers with authentication | ||
| from guidellm.settings import settings |
There was a problem hiding this comment.
Don't late import. This should be in import section at top.
|
|
||
| # Resolve API key (highest priority): explicit param > settings | ||
| resolved_api_key = api_key or settings.openai.api_key | ||
| resolved_bearer_token = bearer_token or settings.openai.bearer_token |
There was a problem hiding this comment.
Can you actually remove all references to bearer_token. I don't think we need both.
There was a problem hiding this comment.
Drop this we have an OCI compliant .containerignore:
https://github.com/vllm-project/guidellm/blob/main/.containerignore
There was a problem hiding this comment.
Drop this we have an OCI compliant Containerfile:
https://github.com/vllm-project/guidellm/blob/main/Containerfile
There was a problem hiding this comment.
Can you move these changes to a new PR? We have had some internal discussion on making arguments to --data more explicit.
2 participants
Summary
This PR adds authentication support to
OpenAIHTTPBackend, enabling GuideLLM to work with OpenAI-compatible servers that require authentication on all endpoints (including/health).Problem
When using GuideLLM with LiteLLM proxy or similar OpenAI-compatible servers that require authentication, the benchmark fails during backend validation because the
/healthendpoint check doesn't include authentication headers:The
OpenAISettingsclass already definesapi_key,bearer_token, andheadersfields, but these were never used in the actual HTTP requests.Solution
api_key,bearer_token, andheadersparameters toOpenAIHTTPBackend.__init__()settings.openaiwhen not explicitly provided/healthendpoint)/v1/modelsendpoint)Usage
Priority Order for Headers
api_key/bearer_token/headersparameterssettings.openai(lowest priority)Testing
/healthendpoint now returns 200 instead of 401Related
This makes the documented example in the docstring actually work: