From 2abe2b6d50d5c0905ccf843fe8c17209c32fd58d Mon Sep 17 00:00:00 2001 From: kayx23 Date: Thu, 23 Jul 2026 13:24:15 +0800 Subject: [PATCH 1/5] docs(ai): document OpenAI protocol detection --- docs/en/latest/plugins/ai-proxy-multi.md | 89 +++++++++++++++++++++--- docs/en/latest/plugins/ai-proxy.md | 74 +++++++++++++++++--- docs/zh/latest/plugins/ai-proxy-multi.md | 89 +++++++++++++++++++++--- docs/zh/latest/plugins/ai-proxy.md | 78 +++++++++++++++++---- 4 files changed, 286 insertions(+), 44 deletions(-) diff --git a/docs/en/latest/plugins/ai-proxy-multi.md b/docs/en/latest/plugins/ai-proxy-multi.md index a91b0f43b160..7e3fe242b57f 100644 --- a/docs/en/latest/plugins/ai-proxy-multi.md +++ b/docs/en/latest/plugins/ai-proxy-multi.md @@ -44,6 +44,20 @@ In addition, the Plugin also supports logging LLM request information in the acc ## Request Format +### OpenAI Request Protocol Detection + +For OpenAI and OpenAI-compatible requests, the Plugin detects the client protocol before selecting the corresponding upstream endpoint: + +| Client protocol | Detection | Route URI | +| --- | --- | --- | +| Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | +| Responses API | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | +| Embeddings | The request body contains `input`, and neither the Responses API nor Chat Completions rule matched. | Any URI matched by the Route. | + +The Plugin checks the rules in table order. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. + +### Chat Completions Request Format + | Name | Type | Required | Description | | ------------------ | ------ | -------- | --------------------------------------------------- | | `messages` | Array | True | An array of message objects. | @@ -76,7 +90,7 @@ When an instance's `provider` is set to `bedrock`, the Plugin expects requests i | balancer.key | string | False | | | Used when `type` is `chash`. When `hash_on` is set to `header` or `cookie`, `key` is required. When `hash_on` is set to `consumer`, `key` is not required as the consumer name will be used as the key automatically. | | instances | array[object] | True | | | LLM instance configurations. | | instances.name | string | True | | | Name of the LLM service instance. | -| instances.provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin will proxy the request to `api.openai.com`. When set to `deepseek`, the Plugin will proxy the request to `api.deepseek.com`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `api.aimlapi.com` by default. When set to `anthropic`, the Plugin will proxy the request to `api.anthropic.com` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `openrouter.ai` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `generativelanguage.googleapis.com` by default. When set to `vertex-ai`, the Plugin will proxy the request to `aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin proxies the request to Amazon Bedrock's Converse API at `bedrock-runtime.{region}.amazonaws.com` and signs the request with AWS SigV4. Requires `provider_conf.region` and `auth.aws`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | +| instances.provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding endpoint on `api.openai.com`. When set to `deepseek`, the Plugin will proxy the request to `api.deepseek.com`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `api.aimlapi.com` by default. When set to `anthropic`, the Plugin will proxy the request to `api.anthropic.com` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `openrouter.ai` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `generativelanguage.googleapis.com` by default. When set to `vertex-ai`, the Plugin will proxy the request to `aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin proxies the request to Amazon Bedrock's Converse API at `bedrock-runtime.{region}.amazonaws.com` and signs the request with AWS SigV4. Requires `provider_conf.region` and `auth.aws`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | instances.provider_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. Required when `provider` is set to `bedrock`. | | instances.provider_conf.project_id | string | True | | | Google Cloud Project ID. | | instances.provider_conf.region | string | True (depending on provider) | | minLength = 1 (for Bedrock) | When `provider` is `vertex-ai`, this is the Google Cloud Region. When `provider` is `bedrock`, this is the AWS region used to construct the Bedrock endpoint and to sign the request with SigV4 (required, must be non-empty). | @@ -2051,6 +2065,70 @@ If `auth.aws.session_token` is set, it is used for temporary credentials (e.g., To enable streaming, send the same Converse request body with `"stream": true`. The Plugin routes the request to Bedrock's `/model//converse-stream` endpoint and forwards each AWS EventStream frame to the client unchanged. The response `Content-Type` is `application/vnd.amazon.eventstream`; clients must parse the binary framing themselves (most AWS SDKs do this automatically). +### Load Balance Responses API Requests + +The following example configures `ai-proxy-multi` to distribute Responses API requests across two OpenAI models. Responses and Embeddings requests both contain `input`, so the Route URI must end in `/v1/responses` for the Plugin to identify the request as Responses. + +Obtain an [OpenAI API key](https://platform.openai.com/api-keys) and save it to an environment variable: + +```shell +export OPENAI_API_KEY= +``` + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "ai-proxy-multi-responses-route", + "uri": "/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy-multi": { + "instances": [ + { + "name": "openai-responses-primary", + "provider": "openai", + "weight": 1, + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1" + } + }, + { + "name": "openai-responses-secondary", + "provider": "openai", + "weight": 1, + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1-mini" + } + } + ] + } + } + }' +``` + +Send a request using the Responses API format: + +```shell +curl "http://127.0.0.1:9080/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Explain API gateways in one sentence." + }' +``` + +The request is forwarded to one of the configured OpenAI instances and the response is returned in the Responses API format. + ### Proxy to Embedding Models The following example demonstrates how you can configure the `ai-proxy-multi` Plugin to proxy requests and load balance between embedding models. @@ -2089,9 +2167,6 @@ curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ }, "options": { "model": "text-embedding-3-small" - }, - "override": { - "endpoint": "https://api.openai.com/v1/embeddings" } }, { @@ -2140,8 +2215,6 @@ services: Authorization: "Bearer ${OPENAI_API_KEY}" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: azure-openai weight: 0 @@ -2193,8 +2266,6 @@ spec: Authorization: "Bearer your-api-key" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: azure-openai weight: 0 @@ -2260,8 +2331,6 @@ spec: Authorization: "Bearer your-api-key" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: azure-openai weight: 0 diff --git a/docs/en/latest/plugins/ai-proxy.md b/docs/en/latest/plugins/ai-proxy.md index 17992d03f6df..211318564893 100644 --- a/docs/en/latest/plugins/ai-proxy.md +++ b/docs/en/latest/plugins/ai-proxy.md @@ -44,6 +44,20 @@ In addition, the Plugin also supports logging LLM request information in the acc ## Request Format +### OpenAI Request Protocol Detection + +For OpenAI and OpenAI-compatible requests, the Plugin detects the client protocol before selecting the corresponding upstream endpoint: + +| Client protocol | Detection | Route URI | +| --- | --- | --- | +| Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | +| Responses API | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | +| Embeddings | The request body contains `input`, and neither the Responses API nor Chat Completions rule matched. | Any URI matched by the Route. | + +The Plugin checks the rules in table order. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. + +### Chat Completions Request Format + | Name | Type | Required | Description | | ------------------ | ------ | -------- | --------------------------------------------------- | | `messages` | Array | True | An array of message objects. | @@ -67,7 +81,7 @@ When `provider` is set to `bedrock`, the Plugin expects requests in the [Bedrock | Name | Type | Required | Default | Valid values | Description | |--------------------|--------|----------|---------|------------------------------------------|-------------| -| provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin will proxy the request to `https://api.openai.com/chat/completions`. When set to `deepseek`, the Plugin will proxy the request to `https://api.deepseek.com/chat/completions`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions` by default. When set to `anthropic`, the Plugin will proxy the request to `https://api.anthropic.com/v1/chat/completions` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://openrouter.ai/api/v1/chat/completions` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` by default. When set to `vertex-ai`, the Plugin will proxy the request to `https://aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin will proxy the request to the AWS Bedrock Converse API (`https://bedrock-runtime..amazonaws.com`) and signs the request with AWS SigV4. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | +| provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding OpenAI endpoint. When set to `deepseek`, the Plugin will proxy the request to `https://api.deepseek.com/chat/completions`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions` by default. When set to `anthropic`, the Plugin will proxy the request to `https://api.anthropic.com/v1/chat/completions` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://openrouter.ai/api/v1/chat/completions` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` by default. When set to `vertex-ai`, the Plugin will proxy the request to `https://aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin will proxy the request to the AWS Bedrock Converse API (`https://bedrock-runtime..amazonaws.com`) and signs the request with AWS SigV4. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | provider_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. Required when `provider` is set to `bedrock`. | | provider_conf.project_id | string | True | | | Google Cloud Project ID. | | provider_conf.region | string | True (depending on provider) | | minLength = 1 (for Bedrock) | When `provider` is `vertex-ai`, this is the Google Cloud Region. When `provider` is `bedrock`, this is the AWS region used to construct the Bedrock endpoint and to sign the request with SigV4 (required, must be non-empty). | @@ -916,9 +930,56 @@ curl "http://127.0.0.1:9080/bedrock/converse" -X POST \ }' --output - ``` +### Proxy OpenAI Responses API + +The following example configures the `ai-proxy` Plugin to proxy OpenAI Responses API requests. Responses and Embeddings requests both contain `input`, so the Route URI must end in `/v1/responses` for the Plugin to identify the request as Responses. + +Obtain an [OpenAI API key](https://platform.openai.com/api-keys) and save it to an environment variable: + +```shell +export OPENAI_API_KEY= +``` + +Create a Route with the `/v1/responses` URI: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "ai-proxy-responses-route", + "uri": "/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy": { + "provider": "openai", + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1" + } + } + } + }' +``` + +Send a request using the Responses API format: + +```shell +curl "http://127.0.0.1:9080/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "input": "Explain API gateways in one sentence." + }' +``` + +The Plugin sends the request to the OpenAI Responses endpoint and returns the response in the Responses API format. To stream the response, add `"stream": true` to the request body and use `curl --no-buffer`. + ### Proxy to OpenAI Embedding Models -The following example demonstrates how you can configure the `ai-proxy` Plugin to proxy requests to embedding models. This example will use the OpenAI embedding model endpoint. +The following example demonstrates how you can configure the `ai-proxy` Plugin to proxy requests to an OpenAI embedding model. The Plugin detects the `input` field and selects the OpenAI Embeddings endpoint. Obtain the OpenAI [API key](https://openai.com/blog/openai-api) and save it to an environment variable: @@ -957,9 +1018,6 @@ curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ "options":{ "model": "text-embedding-3-small", "encoding_format": "float" - }, - "override": { - "endpoint": "https://api.openai.com/v1/embeddings" } } } @@ -990,8 +1048,6 @@ services: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" ``` Synchronize the configuration to the gateway: @@ -1031,8 +1087,6 @@ spec: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute @@ -1086,8 +1140,6 @@ spec: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" ``` diff --git a/docs/zh/latest/plugins/ai-proxy-multi.md b/docs/zh/latest/plugins/ai-proxy-multi.md index 8f3ea99dc615..9a95a1563307 100644 --- a/docs/zh/latest/plugins/ai-proxy-multi.md +++ b/docs/zh/latest/plugins/ai-proxy-multi.md @@ -44,6 +44,20 @@ import TabItem from '@theme/TabItem'; ## 请求格式 +### OpenAI 请求协议检测 + +对于 OpenAI 和 OpenAI 兼容请求,插件会先检测客户端协议,再选择对应的上游端点: + +| 客户端协议 | 检测条件 | 路由 URI | +| --- | --- | --- | +| Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | +| Responses API | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | +| Embeddings | 请求体包含 `input`,并且 Responses API 和 Chat Completions 规则均未匹配。 | 路由匹配的任意 URI。 | + +插件按表中顺序检查规则。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。 + +### Chat Completions 请求格式 + | 名称 | 类型 | 必选项 | 描述 | | ------------------ | ------ | -------- | --------------------------------------------------- | | `messages` | Array | 是 | 消息对象数组。 | @@ -76,7 +90,7 @@ import TabItem from '@theme/TabItem'; | balancer.key | string | 否 | | | 当 `type` 为 `chash` 时使用。当 `hash_on` 设置为 `header` 或 `cookie` 时,需要 `key`。当 `hash_on` 设置为 `consumer` 时,不需要 `key`,因为消费者名称将自动用作键。 | | instances | array[object] | 是 | | | LLM 实例配置。 | | instances.name | string | 是 | | | LLM 服务实例的名称。 | -| instances.provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。设置为 `openai` 时,插件将代理请求到 `api.openai.com`。设置为 `deepseek` 时,插件将代理请求到 `api.deepseek.com`。设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.aimlapi.com`。设置为 `anthropic` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.anthropic.com`。设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `openrouter.ai`。设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `generativelanguage.googleapis.com`。设置为 `vertex-ai` 时,插件默认将请求代理到 `aiplatform.googleapis.com`,且需要配置 `provider_conf` 或 `override`。设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。 | +| instances.provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到 `api.openai.com` 上对应的端点。设置为 `deepseek` 时,插件将代理请求到 `api.deepseek.com`。设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.aimlapi.com`。设置为 `anthropic` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.anthropic.com`。设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `openrouter.ai`。设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `generativelanguage.googleapis.com`。设置为 `vertex-ai` 时,插件默认将请求代理到 `aiplatform.googleapis.com`,且需要配置 `provider_conf` 或 `override`。设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。 | | instances.provider_conf | object | 否 | | | 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必填。当 `provider` 设置为 `bedrock` 时必填。 | | instances.provider_conf.project_id | string | 是 | | | Google Cloud 项目 ID。 | | instances.provider_conf.region | string | 视提供商而定 | | minLength = 1(Bedrock 时) | 当 `provider` 为 `vertex-ai` 时,此项为 Google Cloud 区域。当 `provider` 为 `bedrock` 时,此项为用于构造 Bedrock 端点并使用 SigV4 对请求进行签名的 AWS 区域(必填,不能为空)。 | @@ -1856,6 +1870,70 @@ https://bedrock-runtime.us-east-1.amazonaws.com/model/arn%3Aaws%3Abedrock%3Aus-e 要启用流式响应,请使用相同的 Converse 请求体,并在其中加上 `"stream": true`。插件会将请求路由到 Bedrock 的 `/model//converse-stream` 接口,并将 AWS EventStream 帧原样转发给客户端。响应的 `Content-Type` 为 `application/vnd.amazon.eventstream`,客户端需自行解析二进制帧(多数 AWS SDK 已自动处理)。 +### 对 Responses API 请求进行负载均衡 + +以下示例配置 `ai-proxy-multi`,在两个 OpenAI 模型之间分配 Responses API 请求。Responses 和 Embeddings 请求都包含 `input`,因此路由 URI 必须以 `/v1/responses` 结尾,插件才能将请求识别为 Responses。 + +获取 [OpenAI API 密钥](https://platform.openai.com/api-keys)并保存到环境变量: + +```shell +export OPENAI_API_KEY= +``` + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "ai-proxy-multi-responses-route", + "uri": "/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy-multi": { + "instances": [ + { + "name": "openai-responses-primary", + "provider": "openai", + "weight": 1, + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1" + } + }, + { + "name": "openai-responses-secondary", + "provider": "openai", + "weight": 1, + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1-mini" + } + } + ] + } + } + }' +``` + +使用 Responses API 格式发送请求: + +```shell +curl "http://127.0.0.1:9080/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "input": "用一句话解释 API 网关。" + }' +``` + +请求会被转发到其中一个已配置的 OpenAI 实例,并以 Responses API 格式返回响应。 + ### 代理到嵌入模型 以下示例演示了如何配置 `ai-proxy-multi` 插件以代理请求并在嵌入模型之间进行负载均衡。 @@ -1894,9 +1972,6 @@ curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ }, "options": { "model": "text-embedding-3-small" - }, - "override": { - "endpoint": "https://api.openai.com/v1/embeddings" } }, { @@ -1945,8 +2020,6 @@ services: Authorization: "Bearer ${OPENAI_API_KEY}" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: openai-compatible weight: 0 @@ -1998,8 +2071,6 @@ spec: Authorization: "Bearer your-api-key" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: openai-compatible weight: 0 @@ -2065,8 +2136,6 @@ spec: Authorization: "Bearer your-api-key" options: model: text-embedding-3-small - override: - endpoint: "https://api.openai.com/v1/embeddings" - name: az-openai-instance provider: openai-compatible weight: 0 diff --git a/docs/zh/latest/plugins/ai-proxy.md b/docs/zh/latest/plugins/ai-proxy.md index aa5d0cce92d0..45da3736fad4 100644 --- a/docs/zh/latest/plugins/ai-proxy.md +++ b/docs/zh/latest/plugins/ai-proxy.md @@ -44,6 +44,20 @@ import TabItem from '@theme/TabItem'; ## 请求格式 +### OpenAI 请求协议检测 + +对于 OpenAI 和 OpenAI 兼容请求,插件会先检测客户端协议,再选择对应的上游端点: + +| 客户端协议 | 检测条件 | 路由 URI | +| --- | --- | --- | +| Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | +| Responses API | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | +| Embeddings | 请求体包含 `input`,并且 Responses API 和 Chat Completions 规则均未匹配。 | 路由匹配的任意 URI。 | + +插件按表中顺序检查规则。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。 + +### Chat Completions 请求格式 + | 名称 | 类型 | 必选项 | 描述 | | ------------------ | ------ | -------- | --------------------------------------------------- | | `messages` | Array | 是 | 消息对象数组。 | @@ -67,7 +81,7 @@ import TabItem from '@theme/TabItem'; | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | |--------------------|--------|----------|---------|------------------------------------------|-------------| -| provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。当设置为 `openai` 时,插件将代理请求到 `https://api.openai.com/chat/completions`。当设置为 `deepseek` 时,插件将代理请求到 `https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic` 时,插件将代理请求到 `https://api.anthropic.com/v1/chat/completions`。当设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://openrouter.ai/api/v1/chat/completions`。当设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`。当设置为 `vertex-ai` 时,插件默认将请求代理到 `https://aiplatform.googleapis.com`,需要配置 `provider_conf` 或 `override`。当设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`https://bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。当设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。当设置为 `azure-openai` 时,插件同样将请求代理到 `override` 中配置的自定义端点,并会额外移除用户请求中的 `model` 参数。 | +| provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。当设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到对应的 OpenAI 端点。当设置为 `deepseek` 时,插件将代理请求到 `https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic` 时,插件将代理请求到 `https://api.anthropic.com/v1/chat/completions`。当设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://openrouter.ai/api/v1/chat/completions`。当设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`。当设置为 `vertex-ai` 时,插件默认将请求代理到 `https://aiplatform.googleapis.com`,需要配置 `provider_conf` 或 `override`。当设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`https://bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。当设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。当设置为 `azure-openai` 时,插件同样将请求代理到 `override` 中配置的自定义端点,并会额外移除用户请求中的 `model` 参数。 | | provider_conf | object | 否 | | | 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必填。当 `provider` 设置为 `bedrock` 时必填。 | | provider_conf.project_id | string | 是 | | | Google Cloud 项目 ID。 | | provider_conf.region | string | 视提供商而定 | | minLength = 1(Bedrock 时) | 当 `provider` 为 `vertex-ai` 时,此项为 Google Cloud 区域。当 `provider` 为 `bedrock` 时,此项为用于构造 Bedrock 端点并使用 SigV4 对请求进行签名的 AWS 区域(必填,不能为空)。 | @@ -911,9 +925,56 @@ curl "http://127.0.0.1:9080/bedrock/converse" -X POST \ }' --output - ``` +### 代理 OpenAI Responses API + +以下示例配置 `ai-proxy` 插件以代理 OpenAI Responses API 请求。Responses 和 Embeddings 请求都包含 `input`,因此路由 URI 必须以 `/v1/responses` 结尾,插件才能将请求识别为 Responses。 + +获取 [OpenAI API 密钥](https://platform.openai.com/api-keys)并保存到环境变量: + +```shell +export OPENAI_API_KEY= +``` + +创建 URI 为 `/v1/responses` 的路由: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "id": "ai-proxy-responses-route", + "uri": "/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy": { + "provider": "openai", + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + }, + "options": { + "model": "gpt-4.1" + } + } + } + }' +``` + +使用 Responses API 格式发送请求: + +```shell +curl "http://127.0.0.1:9080/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "input": "用一句话解释 API 网关。" + }' +``` + +插件会将请求发送到 OpenAI Responses 端点,并以 Responses API 格式返回响应。如需流式响应,请在请求体中添加 `"stream": true`,并使用 `curl --no-buffer`。 + ### 代理到 OpenAI 嵌入模型 -以下示例演示了如何配置 `ai-proxy` 插件以将请求代理到嵌入模型。此示例将使用 OpenAI 嵌入模型端点。 +以下示例演示如何配置 `ai-proxy` 插件以代理对 OpenAI 嵌入模型的请求。插件会检测 `input` 字段并选择 OpenAI Embeddings 端点。 获取 OpenAI [API 密钥](https://openai.com/blog/openai-api)并保存到环境变量: @@ -932,7 +993,7 @@ values={[ -创建路由并配置 `ai-proxy` 插件,将 `provider` 设置为 `openai`,指定嵌入模型名称,添加 `encoding_format` 参数以配置返回的嵌入向量为浮点数列表,并使用 `override` 将默认端点覆盖为 [嵌入 API 端点](https://platform.openai.com/docs/api-reference/embeddings): +创建路由并配置 `ai-proxy` 插件,将 `provider` 设置为 `openai`,指定嵌入模型名称,并添加 `encoding_format` 参数,将返回的嵌入向量配置为浮点数列表: ```shell curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ @@ -952,9 +1013,6 @@ curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ "options":{ "model": "text-embedding-3-small", "encoding_format": "float" - }, - "override": { - "endpoint": "https://api.openai.com/v1/embeddings" } } } @@ -965,7 +1023,7 @@ curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -创建包含 `ai-proxy` 插件配置的路由,将 `provider` 设置为 `openai`,指定嵌入模型名称,添加 `encoding_format` 参数,并使用 `override` 将默认端点覆盖为 [嵌入 API 端点](https://platform.openai.com/docs/api-reference/embeddings): +创建包含 `ai-proxy` 插件配置的路由,将 `provider` 设置为 `openai`,指定嵌入模型名称,并添加 `encoding_format` 参数: ```yaml title="adc.yaml" services: @@ -985,8 +1043,6 @@ services: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" ``` 将配置同步到网关: @@ -1026,8 +1082,6 @@ spec: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute @@ -1081,8 +1135,6 @@ spec: options: model: text-embedding-3-small encoding_format: float - override: - endpoint: "https://api.openai.com/v1/embeddings" ``` From e8131736eff2b54186af4549501e3d0168a64135 Mon Sep 17 00:00:00 2001 From: kayx23 Date: Thu, 23 Jul 2026 13:28:07 +0800 Subject: [PATCH 2/5] docs(ai-request-rewrite): clarify LLM request format --- docs/en/latest/plugins/ai-request-rewrite.md | 4 +++- docs/zh/latest/plugins/ai-request-rewrite.md | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/en/latest/plugins/ai-request-rewrite.md b/docs/en/latest/plugins/ai-request-rewrite.md index df518598d5f1..b689d7091abe 100644 --- a/docs/en/latest/plugins/ai-request-rewrite.md +++ b/docs/en/latest/plugins/ai-request-rewrite.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; The `ai-request-rewrite` Plugin processes client requests by forwarding them to LLM services for transformation before relaying them to Upstream services. This enables LLM-powered modifications such as data redaction, content enrichment, or reformatting. The Plugin supports integration with OpenAI, DeepSeek, Gemini, Vertex AI, Anthropic, OpenRouter, and other OpenAI-compatible APIs. +The LLM call used for rewriting is separate from the client's request format. With the `openai` provider, the Plugin builds a non-streaming Chat Completions request from the configured prompt and the original request body. It does not use the Chat Completions, Responses API, or Embeddings detection rules of [`ai-proxy`](./ai-proxy.md#openai-request-protocol-detection). + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | @@ -47,7 +49,7 @@ The `ai-request-rewrite` Plugin processes client requests by forwarding them to | `auth` | object | True | | | Authentication configurations. | | `auth.header` | object | False | | | Authentication headers. Key must match pattern `^[a-zA-Z0-9._-]+$`. At least one of `header` and `query` should be configured. | | `auth.query` | object | False | | | Authentication query parameters. Key must match pattern `^[a-zA-Z0-9._-]+$`. At least one of `header` and `query` should be configured. | -| `options` | object | False | | | Model configurations. In addition to `model`, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI, you can configure additional parameters such as `temperature`, `top_p`, and `stream`. See your LLM provider's API documentation for more available options. | +| `options` | object | False | | | Model configurations. In addition to `model`, you can configure non-streaming generation parameters, such as `temperature` and `top_p`, which are forwarded to the upstream LLM service. The Plugin must receive a complete JSON response before it can replace the original request body. | | `options.model` | string | False | | | Name of the LLM model, such as `gpt-4` or `gpt-3.5`. See your LLM provider's API documentation for more available models. | | `override` | object | False | | | Override setting. | | `override.endpoint` | string | False | | | LLM provider endpoint. Required when `provider` is `openai-compatible`. | diff --git a/docs/zh/latest/plugins/ai-request-rewrite.md b/docs/zh/latest/plugins/ai-request-rewrite.md index bd0c1c0585c3..f9eb89c92b95 100644 --- a/docs/zh/latest/plugins/ai-request-rewrite.md +++ b/docs/zh/latest/plugins/ai-request-rewrite.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; `ai-request-rewrite` 插件在将客户端请求转发到上游服务之前,先将请求发送到 LLM 服务进行转换处理。这使得 LLM 能够对请求进行数据脱敏、内容增强或格式转换等修改。该插件支持集成 OpenAI、DeepSeek、Gemini、Vertex AI、Anthropic、OpenRouter 以及其他 OpenAI 兼容的 API。 +用于重写的 LLM 调用与客户端请求格式相互独立。使用 `openai` 提供商时,插件会根据配置的提示词和原始请求体构建非流式 Chat Completions 请求。它不会使用 [`ai-proxy`](./ai-proxy.md#openai-请求协议检测) 的 Chat Completions、Responses API 或 Embeddings 检测规则。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | @@ -47,7 +49,7 @@ import TabItem from '@theme/TabItem'; | `auth` | object | 是 | | | 身份验证配置。 | | `auth.header` | object | 否 | | | 身份验证请求头。键必须匹配模式 `^[a-zA-Z0-9._-]+$`。`header` 和 `query` 至少需要配置其中一个。 | | `auth.query` | object | 否 | | | 身份验证查询参数。键必须匹配模式 `^[a-zA-Z0-9._-]+$`。`header` 和 `query` 至少需要配置其中一个。 | -| `options` | object | 否 | | | 模型配置。除了 `model` 之外,还可以配置其他参数,这些参数会在请求体中转发给上游 LLM 服务。例如,使用 OpenAI 时,可以配置 `temperature`、`top_p` 和 `stream` 等参数。更多可用选项请参阅 LLM 提供商的 API 文档。 | +| `options` | object | 否 | | | 模型配置。除了 `model` 之外,还可以配置 `temperature` 和 `top_p` 等非流式生成参数,这些参数会转发给上游 LLM 服务。插件必须收到完整的 JSON 响应,才能替换原始请求体。 | | `options.model` | string | 否 | | | LLM 模型名称,例如 `gpt-4` 或 `gpt-3.5`。更多可用模型请参阅 LLM 提供商的 API 文档。 | | `override` | object | 否 | | | 覆盖设置。 | | `override.endpoint` | string | 否 | | | LLM 提供商端点。当 `provider` 为 `openai-compatible` 时必填。 | From e88cee8fe62b4623188252c759ddd4483eaff5d2 Mon Sep 17 00:00:00 2001 From: kayx23 Date: Thu, 23 Jul 2026 13:31:48 +0800 Subject: [PATCH 3/5] docs(ai): clarify protocol-aware plugin behavior --- docs/en/latest/plugins/ai-aliyun-content-moderation.md | 2 ++ docs/en/latest/plugins/ai-aws-content-moderation.md | 2 +- docs/en/latest/plugins/ai-cache.md | 2 ++ docs/en/latest/plugins/ai-lakera-guard.md | 6 ++++-- docs/en/latest/plugins/ai-prompt-decorator.md | 2 ++ docs/en/latest/plugins/ai-prompt-guard.md | 2 ++ docs/en/latest/plugins/ai-prompt-template.md | 2 ++ docs/en/latest/plugins/ai-rag.md | 2 ++ docs/zh/latest/plugins/ai-aliyun-content-moderation.md | 2 ++ docs/zh/latest/plugins/ai-aws-content-moderation.md | 2 +- docs/zh/latest/plugins/ai-cache.md | 2 ++ docs/zh/latest/plugins/ai-lakera-guard.md | 6 ++++-- docs/zh/latest/plugins/ai-prompt-decorator.md | 2 ++ docs/zh/latest/plugins/ai-prompt-guard.md | 2 ++ docs/zh/latest/plugins/ai-prompt-template.md | 2 ++ docs/zh/latest/plugins/ai-rag.md | 2 ++ 16 files changed, 34 insertions(+), 6 deletions(-) diff --git a/docs/en/latest/plugins/ai-aliyun-content-moderation.md b/docs/en/latest/plugins/ai-aliyun-content-moderation.md index 89c0f20c62f3..9f3ee4cfaf8a 100644 --- a/docs/en/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/en/latest/plugins/ai-aliyun-content-moderation.md @@ -45,6 +45,8 @@ Please ensure that the `access_key_secret` is correctly configured in the Plugin The `ai-aliyun-content-moderation` Plugin should be used with either [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. +The Plugin supports Chat Completions, Responses API, and Embeddings requests. For Responses, it moderates `input` and returns blocked non-streaming or streaming requests in Responses API format. For Embeddings, it moderates a string or an array of strings in `input` and returns rejected requests as OpenAI-style errors. + ## Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-aws-content-moderation.md b/docs/en/latest/plugins/ai-aws-content-moderation.md index 063911ddd6d9..d05854b4bef6 100644 --- a/docs/en/latest/plugins/ai-aws-content-moderation.md +++ b/docs/en/latest/plugins/ai-aws-content-moderation.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-aws-content-moderation` Plugin integrates with [AWS Comprehend](https://aws.amazon.com/comprehend/) to check request content for toxicity when proxying to LLMs, such as profanity, hate speech, insult, harassment, violence, and more, rejecting requests if the evaluated outcome exceeds the configured threshold. -The Plugin is protocol-aware: it extracts the prompt content from the LLM request (for example `messages[].content`) and moderates only that decoded text, rather than the raw request body. +The Plugin supports detected Chat Completions, Responses API, and Embeddings requests. It extracts text from `messages`, Responses `instructions` and `input`, or Embeddings `input`, respectively, and moderates only that decoded text rather than the raw request body. Rejected requests use the detected protocol's response format. The `ai-aws-content-moderation` Plugin should be used with either [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. diff --git a/docs/en/latest/plugins/ai-cache.md b/docs/en/latest/plugins/ai-cache.md index 907b8d177719..d399d5b83f7e 100644 --- a/docs/en/latest/plugins/ai-cache.md +++ b/docs/en/latest/plugins/ai-cache.md @@ -45,6 +45,8 @@ This Plugin supports two cache layers: - **Exact (L1):** A SHA-256 fingerprint of the effective prompt is used as the Redis key. An identical prompt always hits the same entry. - **Semantic (L2):** When L1 misses, the prompt is embedded into a vector and a nearest-neighbour search retrieves a past response whose embedding is within the configured similarity threshold. L2 is disabled by default; enable it by adding `"semantic"` to `layers`. +Exact caching supports detected Chat Completions, Responses API, and Embeddings requests and keeps the protocols in separate cache entries. Semantic caching applies only to Chat Completions requests; Responses and Embeddings requests can use the exact layer but bypass the semantic layer. + The `ai-cache` Plugin must be used with the [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin. ### Streaming diff --git a/docs/en/latest/plugins/ai-lakera-guard.md b/docs/en/latest/plugins/ai-lakera-guard.md index c43a432aa6f6..1d7001e7d242 100644 --- a/docs/en/latest/plugins/ai-lakera-guard.md +++ b/docs/en/latest/plugins/ai-lakera-guard.md @@ -43,7 +43,9 @@ The `ai-lakera-guard` Plugin integrates with the [Lakera Guard API (v2)](https:/ Which detectors run and at what thresholds are controlled entirely by the **Lakera project policy**, selected with `project_id`. There is no gateway-side detector list; Lakera returns a single verdict per call. -The `ai-lakera-guard` Plugin should be used with either the [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. It relies on the context that `ai-proxy` populates to extract chat content in a protocol-aware way. +The `ai-lakera-guard` Plugin should be used with either the [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. It relies on the context that `ai-proxy` populates to extract request content in a protocol-aware way. + +Request scanning supports Chat Completions, Responses API, and Embeddings requests. For Responses, the Plugin converts `instructions` and text from `input` into conversation messages. For Embeddings, it scans text from `input`. Response scanning applies to protocols that return generated text, including Chat Completions and Responses; Embeddings responses contain vectors rather than text. Requests that did not pass through `ai-proxy`/`ai-proxy-multi` (for example plain HTTP traffic when the Plugin is bound at the Consumer or Service level) cannot be inspected. By default such requests are passed through unchecked; this is configurable via `fail_mode`. @@ -63,7 +65,7 @@ The Plugin can scan the request prompt (`direction: input`), the LLM response (` | timeout | integer | False | `5000` | >= 1 | Lakera request timeout in milliseconds. | | ssl_verify | boolean | False | `true` | | If `true`, verify the TLS certificate of the Lakera endpoint. | | reveal_failure_categories | boolean | False | `false` | | If `true`, append the matched Lakera `detector_type`s (with their confidence result) to the deny message returned to the client. The full per-detector `breakdown` is always requested from Lakera and written to the gateway logs regardless of this setting; this flag only controls client-facing exposure. | -| deny_code | integer | False | `200` | 200 - 599 | HTTP status code returned when a request is blocked. Defaults to `200` so the body — a provider-compatible chat completion (or SSE) carrying `request_failure_message` — parses as a normal refusal in client SDKs (matching how Lakera Guard itself returns `200` with a verdict). Set a 4xx (e.g. `403`) if you prefer blocks to surface as HTTP errors. | +| deny_code | integer | False | `200` | 200 - 599 | HTTP status code returned when a request is blocked. Defaults to `200` so the body in the detected protocol's format carrying `request_failure_message` parses as a normal refusal in client SDKs (matching how Lakera Guard itself returns `200` with a verdict). Set a 4xx (e.g. `403`) if you prefer blocks to surface as HTTP errors. | | request_failure_message | string | False | `Request blocked by Lakera Guard` | | Refusal text returned (as the assistant message of a provider-compatible response) when a request is blocked. | | response_failure_message | string | False | `Response blocked by Lakera Guard` | | Refusal text returned (as the assistant message of a provider-compatible response) when an LLM response is blocked (`direction` `output` or `both`). | diff --git a/docs/en/latest/plugins/ai-prompt-decorator.md b/docs/en/latest/plugins/ai-prompt-decorator.md index 454392278f34..563871e47b47 100644 --- a/docs/en/latest/plugins/ai-prompt-decorator.md +++ b/docs/en/latest/plugins/ai-prompt-decorator.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; The `ai-prompt-decorator` Plugin modifies user input prompts by prefixing and appending pre-engineered prompts to set contexts in content generation. This practice helps the model operate within desired guidelines during interactions. +The Plugin supports Chat Completions and Responses API requests. For Responses, content from `prepend` is added to `instructions`, and content from `append` is added to `input`. The Plugin does not modify Embeddings requests because that format has no prompt roles to prepend or append. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-prompt-guard.md b/docs/en/latest/plugins/ai-prompt-guard.md index b5ebd17cca63..44546510ad71 100644 --- a/docs/en/latest/plugins/ai-prompt-guard.md +++ b/docs/en/latest/plugins/ai-prompt-guard.md @@ -40,6 +40,8 @@ The `ai-prompt-guard` Plugin safeguards your LLM endpoints by inspecting and val When both `allow_patterns` and `deny_patterns` are configured, the Plugin first ensures that at least one `allow_patterns` is matched. If none match, the request is rejected. If an allowed pattern is matched, it then checks for any occurrences of denied patterns. +The Plugin supports Chat Completions and Responses API requests. For Responses, it checks user content in `input` and, when `match_all_roles` is `true`, system content in `instructions`. For Embeddings, it checks `input` when it is a string; an array of input strings is not inspected. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-prompt-template.md b/docs/en/latest/plugins/ai-prompt-template.md index 2d3e80de8ddd..b6b3a42847e8 100644 --- a/docs/en/latest/plugins/ai-prompt-template.md +++ b/docs/en/latest/plugins/ai-prompt-template.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; The `ai-prompt-template` Plugin supports pre-configuring prompt templates that only accept user inputs in designated template variables, in a "fill in the blank" fashion. It simplifies access to LLM providers, such as OpenAI and Anthropic, by letting you define reusable prompt structures. +The Plugin generates a Chat Completions request containing a `messages` array. It does not generate Responses API or Embeddings requests. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-rag.md b/docs/en/latest/plugins/ai-rag.md index fe1130ddffb4..7e351ffe7a99 100644 --- a/docs/en/latest/plugins/ai-rag.md +++ b/docs/en/latest/plugins/ai-rag.md @@ -42,6 +42,8 @@ The `ai-rag` Plugin provides Retrieval-Augmented Generation (RAG) capabilities w The Plugin supports using [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) and [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) services for generating embeddings and performing vector search. PRs for introducing support for other service providers are welcomed. +The Plugin enriches Chat Completions and Responses API requests. For Responses, it appends retrieved context to `input`. It does not enrich direct Embeddings API requests. The nested `ai_rag.embeddings` object configures the embedding input used internally for retrieval and is distinct from a top-level Embeddings request. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md index ae420dbc1f76..9e5dc48aa079 100644 --- a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md @@ -45,6 +45,8 @@ import TabItem from '@theme/TabItem'; `ai-aliyun-content-moderation` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件配合使用以代理 LLM 请求。 +该插件支持 Chat Completions、Responses API 和 Embeddings 请求。对于 Responses,它会审核 `input`,并以 Responses API 格式返回被拦截的非流式或流式请求。对于 Embeddings,它会审核 `input` 中的字符串或字符串数组,并以 OpenAI 风格的错误响应返回被拒绝的请求。 + ## 属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-aws-content-moderation.md b/docs/zh/latest/plugins/ai-aws-content-moderation.md index d98259984e4f..27a83cd28664 100644 --- a/docs/zh/latest/plugins/ai-aws-content-moderation.md +++ b/docs/zh/latest/plugins/ai-aws-content-moderation.md @@ -40,7 +40,7 @@ import TabItem from '@theme/TabItem'; `ai-aws-content-moderation` 插件集成了 [AWS Comprehend](https://aws.amazon.com/comprehend/),用于在代理请求到 LLM 时检查请求内容中的有害内容,例如亵渎、仇恨言论、侮辱、骚扰、暴力等,如果评估结果超过配置的阈值则拒绝请求。 -该插件是协议感知的:它会从 LLM 请求中提取提示内容(例如 `messages[].content`),仅审核解码后的文本,而不是原始请求体。 +该插件支持检测到的 Chat Completions、Responses API 和 Embeddings 请求。它会分别从 `messages`、Responses 的 `instructions` 和 `input`,或 Embeddings 的 `input` 中提取文本,并仅审核解码后的文本,而不是原始请求体。被拒绝的请求使用检测到的协议对应的响应格式。 `ai-aws-content-moderation` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件一起使用,以代理 LLM 请求。 diff --git a/docs/zh/latest/plugins/ai-cache.md b/docs/zh/latest/plugins/ai-cache.md index 5c9e51234859..3437e77ac22f 100644 --- a/docs/zh/latest/plugins/ai-cache.md +++ b/docs/zh/latest/plugins/ai-cache.md @@ -45,6 +45,8 @@ import TabItem from '@theme/TabItem'; - **精确缓存(L1):** 对有效提示词计算 SHA-256 指纹并用作 Redis 键。完全相同的提示词始终命中同一条缓存条目。 - **语义缓存(L2):** 当 L1 未命中时,将提示词向量化,并通过最近邻搜索检索相似度在阈值以上的历史响应。L2 默认关闭;在 `layers` 中加入 `"semantic"` 即可启用。 +精确缓存支持检测到的 Chat Completions、Responses API 和 Embeddings 请求,并将不同协议存储在独立的缓存条目中。语义缓存仅适用于 Chat Completions 请求;Responses 和 Embeddings 请求可以使用精确缓存层,但会绕过语义缓存层。 + `ai-cache` 插件必须与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件一起使用。 ### 流式响应 diff --git a/docs/zh/latest/plugins/ai-lakera-guard.md b/docs/zh/latest/plugins/ai-lakera-guard.md index 46750da4d637..75c5dc05af3f 100644 --- a/docs/zh/latest/plugins/ai-lakera-guard.md +++ b/docs/zh/latest/plugins/ai-lakera-guard.md @@ -43,7 +43,9 @@ import TabItem from '@theme/TabItem'; 运行哪些检测器以及使用何种阈值,完全由通过 `project_id` 选择的 **Lakera 项目策略**控制。网关侧没有检测器列表;Lakera 每次调用返回单一的判定结果。 -`ai-lakera-guard` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件配合使用以代理 LLM 请求。它依赖 `ai-proxy` 填充的上下文,以协议感知的方式提取对话内容。 +`ai-lakera-guard` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件配合使用以代理 LLM 请求。它依赖 `ai-proxy` 填充的上下文,以协议感知的方式提取请求内容。 + +请求扫描支持 Chat Completions、Responses API 和 Embeddings 请求。对于 Responses,插件会将 `instructions` 和 `input` 中的文本转换为对话消息。对于 Embeddings,插件会扫描 `input` 中的文本。响应扫描适用于返回生成文本的协议,包括 Chat Completions 和 Responses;Embeddings 响应包含向量而不是文本。 未经过 `ai-proxy`/`ai-proxy-multi` 的请求(例如插件绑定在 Consumer 或 Service 级别时的普通 HTTP 流量)无法被检查。默认情况下,此类请求会被直接放行而不做检查;该行为可通过 `fail_mode` 配置。 @@ -63,7 +65,7 @@ import TabItem from '@theme/TabItem'; | timeout | integer | 否 | `5000` | >= 1 | Lakera 请求超时时间(毫秒)。 | | ssl_verify | boolean | 否 | `true` | | 如果为 `true`,则验证 Lakera 端点的 TLS 证书。 | | reveal_failure_categories | boolean | 否 | `false` | | 如果为 `true`,将匹配到的 Lakera `detector_type`(及其置信度结果)追加到返回给客户端的拒绝消息中。无论该设置如何,插件始终会向 Lakera 请求完整的每个检测器的 `breakdown` 并写入网关日志;此标志仅控制面向客户端的暴露。 | -| deny_code | integer | 否 | `200` | 200 - 599 | 请求被拦截时返回的 HTTP 状态码。默认为 `200`,使响应体——一个携带 `request_failure_message` 的、与提供商兼容的聊天补全(或 SSE)——在客户端 SDK 中被解析为正常的拒绝消息(与 Lakera Guard 自身返回 `200` 并附带判定结果的方式一致)。如果你希望拦截以 HTTP 错误的形式呈现,可设置为 4xx(例如 `403`)。 | +| deny_code | integer | 否 | `200` | 200 - 599 | 请求被拦截时返回的 HTTP 状态码。默认为 `200`,使携带 `request_failure_message` 且采用检测到的协议格式的响应体在客户端 SDK 中被解析为正常的拒绝消息(与 Lakera Guard 自身返回 `200` 并附带判定结果的方式一致)。如果你希望拦截以 HTTP 错误的形式呈现,可设置为 4xx(例如 `403`)。 | | request_failure_message | string | 否 | `Request blocked by Lakera Guard` | | 请求被拦截时返回的拒绝文本(作为与提供商兼容的响应中的 assistant 消息)。 | | response_failure_message | string | 否 | `Response blocked by Lakera Guard` | | LLM 响应被拦截时(`direction` 为 `output` 或 `both`)返回的拒绝文本(作为与提供商兼容的响应中的 assistant 消息)。 | diff --git a/docs/zh/latest/plugins/ai-prompt-decorator.md b/docs/zh/latest/plugins/ai-prompt-decorator.md index bd32a5a2b803..40c70d12ef80 100644 --- a/docs/zh/latest/plugins/ai-prompt-decorator.md +++ b/docs/zh/latest/plugins/ai-prompt-decorator.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; `ai-prompt-decorator` 插件通过在用户输入提示前后添加预设提示来修改用户输入,以在内容生成中设定上下文。这种做法有助于模型在交互过程中按照预期的指导方针运行。 +该插件支持 Chat Completions 和 Responses API 请求。对于 Responses,`prepend` 中的内容会添加到 `instructions`,`append` 中的内容会添加到 `input`。该插件不会修改 Embeddings 请求,因为该格式没有可前置或追加的提示角色。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-prompt-guard.md b/docs/zh/latest/plugins/ai-prompt-guard.md index 80e177b0a787..32f24abdd7d9 100644 --- a/docs/zh/latest/plugins/ai-prompt-guard.md +++ b/docs/zh/latest/plugins/ai-prompt-guard.md @@ -40,6 +40,8 @@ import TabItem from '@theme/TabItem'; 当同时配置了 `allow_patterns` 和 `deny_patterns` 时,插件首先确保至少匹配一个 `allow_patterns`。如果没有匹配,请求将被拒绝。如果匹配了允许的模式,它会继续检查是否存在任何拒绝模式的匹配。 +该插件支持 Chat Completions 和 Responses API 请求。对于 Responses,它会检查 `input` 中的用户内容;当 `match_all_roles` 为 `true` 时,还会检查 `instructions` 中的系统内容。对于 Embeddings,它会在 `input` 为字符串时进行检查;字符串数组形式的输入不会被检查。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-prompt-template.md b/docs/zh/latest/plugins/ai-prompt-template.md index 60617e8bf7c2..13247f2e4aae 100644 --- a/docs/zh/latest/plugins/ai-prompt-template.md +++ b/docs/zh/latest/plugins/ai-prompt-template.md @@ -38,6 +38,8 @@ import TabItem from '@theme/TabItem'; `ai-prompt-template` 插件支持预先配置提示词模板,这些模板仅接受用户在指定的模板变量中输入,采用"填空"的方式。它简化了对 OpenAI、Anthropic 等大语言模型提供商及其模型的访问,让你可以定义可复用的提示词结构。 +该插件会生成包含 `messages` 数组的 Chat Completions 请求。它不会生成 Responses API 或 Embeddings 请求。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-rag.md b/docs/zh/latest/plugins/ai-rag.md index 7adffdbf2f59..60195d2c27d1 100644 --- a/docs/zh/latest/plugins/ai-rag.md +++ b/docs/zh/latest/plugins/ai-rag.md @@ -42,6 +42,8 @@ import TabItem from '@theme/TabItem'; 该插件支持使用 [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) 和 [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) 服务来生成嵌入和执行向量搜索。欢迎提交 PR 以引入对其他服务提供商的支持。 +该插件会增强 Chat Completions 和 Responses API 请求。对于 Responses,它会将检索到的上下文追加到 `input`。它不会增强直接的 Embeddings API 请求。嵌套的 `ai_rag.embeddings` 对象用于配置检索过程中内部使用的嵌入输入,与顶层 Embeddings 请求不同。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | From e007ee9aa342bbc06ccb60fd8cd7d88d413a74fe Mon Sep 17 00:00:00 2001 From: kayx23 Date: Thu, 23 Jul 2026 13:43:20 +0800 Subject: [PATCH 4/5] docs(ai): document all request protocols --- .../latest/plugins/ai-aliyun-content-moderation.md | 2 +- .../en/latest/plugins/ai-aws-content-moderation.md | 2 +- docs/en/latest/plugins/ai-cache.md | 2 +- docs/en/latest/plugins/ai-lakera-guard.md | 2 +- docs/en/latest/plugins/ai-prompt-decorator.md | 2 +- docs/en/latest/plugins/ai-prompt-guard.md | 2 +- docs/en/latest/plugins/ai-proxy-multi.md | 14 ++++++++------ docs/en/latest/plugins/ai-proxy.md | 14 ++++++++------ docs/en/latest/plugins/ai-rag.md | 2 +- docs/en/latest/plugins/ai-request-rewrite.md | 2 +- .../latest/plugins/ai-aliyun-content-moderation.md | 2 +- .../zh/latest/plugins/ai-aws-content-moderation.md | 2 +- docs/zh/latest/plugins/ai-cache.md | 2 +- docs/zh/latest/plugins/ai-lakera-guard.md | 2 +- docs/zh/latest/plugins/ai-prompt-decorator.md | 2 +- docs/zh/latest/plugins/ai-prompt-guard.md | 2 +- docs/zh/latest/plugins/ai-proxy-multi.md | 14 ++++++++------ docs/zh/latest/plugins/ai-proxy.md | 14 ++++++++------ docs/zh/latest/plugins/ai-rag.md | 2 +- docs/zh/latest/plugins/ai-request-rewrite.md | 2 +- 20 files changed, 48 insertions(+), 40 deletions(-) diff --git a/docs/en/latest/plugins/ai-aliyun-content-moderation.md b/docs/en/latest/plugins/ai-aliyun-content-moderation.md index 9f3ee4cfaf8a..24c30852b445 100644 --- a/docs/en/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/en/latest/plugins/ai-aliyun-content-moderation.md @@ -45,7 +45,7 @@ Please ensure that the `access_key_secret` is correctly configured in the Plugin The `ai-aliyun-content-moderation` Plugin should be used with either [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. -The Plugin supports Chat Completions, Responses API, and Embeddings requests. For Responses, it moderates `input` and returns blocked non-streaming or streaming requests in Responses API format. For Embeddings, it moderates a string or an array of strings in `input` and returns rejected requests as OpenAI-style errors. +The Plugin moderates Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests using each protocol's native content structure. For Responses, it moderates `input` and returns blocked non-streaming or streaming requests in Responses API format. For Embeddings, it moderates a string or an array of strings in `input` and returns rejected requests as OpenAI-style errors. ## Attributes diff --git a/docs/en/latest/plugins/ai-aws-content-moderation.md b/docs/en/latest/plugins/ai-aws-content-moderation.md index d05854b4bef6..ff7ed5753d4f 100644 --- a/docs/en/latest/plugins/ai-aws-content-moderation.md +++ b/docs/en/latest/plugins/ai-aws-content-moderation.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-aws-content-moderation` Plugin integrates with [AWS Comprehend](https://aws.amazon.com/comprehend/) to check request content for toxicity when proxying to LLMs, such as profanity, hate speech, insult, harassment, violence, and more, rejecting requests if the evaluated outcome exceeds the configured threshold. -The Plugin supports detected Chat Completions, Responses API, and Embeddings requests. It extracts text from `messages`, Responses `instructions` and `input`, or Embeddings `input`, respectively, and moderates only that decoded text rather than the raw request body. Rejected requests use the detected protocol's response format. +The Plugin supports Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests. It extracts text using the detected protocol's native content structure and moderates only that decoded text rather than the raw request body. Rejected requests use the detected protocol's response format. The `ai-aws-content-moderation` Plugin should be used with either [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. diff --git a/docs/en/latest/plugins/ai-cache.md b/docs/en/latest/plugins/ai-cache.md index d399d5b83f7e..27cb2790d1c7 100644 --- a/docs/en/latest/plugins/ai-cache.md +++ b/docs/en/latest/plugins/ai-cache.md @@ -45,7 +45,7 @@ This Plugin supports two cache layers: - **Exact (L1):** A SHA-256 fingerprint of the effective prompt is used as the Redis key. An identical prompt always hits the same entry. - **Semantic (L2):** When L1 misses, the prompt is embedded into a vector and a nearest-neighbour search retrieves a past response whose embedding is within the configured similarity threshold. L2 is disabled by default; enable it by adding `"semantic"` to `layers`. -Exact caching supports detected Chat Completions, Responses API, and Embeddings requests and keeps the protocols in separate cache entries. Semantic caching applies only to Chat Completions requests; Responses and Embeddings requests can use the exact layer but bypass the semantic layer. +Exact caching supports Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests and keeps the protocols in separate cache entries. Semantic caching applies only to Chat Completions requests; the other protocols can use the exact layer but bypass the semantic layer. The `ai-cache` Plugin must be used with the [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin. diff --git a/docs/en/latest/plugins/ai-lakera-guard.md b/docs/en/latest/plugins/ai-lakera-guard.md index 1d7001e7d242..7557571f4404 100644 --- a/docs/en/latest/plugins/ai-lakera-guard.md +++ b/docs/en/latest/plugins/ai-lakera-guard.md @@ -45,7 +45,7 @@ Which detectors run and at what thresholds are controlled entirely by the **Lake The `ai-lakera-guard` Plugin should be used with either the [`ai-proxy`](./ai-proxy.md) or [`ai-proxy-multi`](./ai-proxy-multi.md) Plugin for proxying LLM requests. It relies on the context that `ai-proxy` populates to extract request content in a protocol-aware way. -Request scanning supports Chat Completions, Responses API, and Embeddings requests. For Responses, the Plugin converts `instructions` and text from `input` into conversation messages. For Embeddings, it scans text from `input`. Response scanning applies to protocols that return generated text, including Chat Completions and Responses; Embeddings responses contain vectors rather than text. +Request scanning supports Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests. For Responses, the Plugin converts `instructions` and text from `input` into conversation messages. For Embeddings, it scans text from `input`. Response scanning applies to the protocols that return generated text; Embeddings responses contain vectors rather than text. Requests that did not pass through `ai-proxy`/`ai-proxy-multi` (for example plain HTTP traffic when the Plugin is bound at the Consumer or Service level) cannot be inspected. By default such requests are passed through unchecked; this is configurable via `fail_mode`. diff --git a/docs/en/latest/plugins/ai-prompt-decorator.md b/docs/en/latest/plugins/ai-prompt-decorator.md index 563871e47b47..d30f18c7c789 100644 --- a/docs/en/latest/plugins/ai-prompt-decorator.md +++ b/docs/en/latest/plugins/ai-prompt-decorator.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-prompt-decorator` Plugin modifies user input prompts by prefixing and appending pre-engineered prompts to set contexts in content generation. This practice helps the model operate within desired guidelines during interactions. -The Plugin supports Chat Completions and Responses API requests. For Responses, content from `prepend` is added to `instructions`, and content from `append` is added to `input`. The Plugin does not modify Embeddings requests because that format has no prompt roles to prepend or append. +The Plugin decorates Chat Completions, Responses API, Anthropic Messages, and Bedrock Converse requests using each protocol's native prompt structure. For Responses, content from `prepend` is added to `instructions`, and content from `append` is added to `input`. The Plugin does not modify Embeddings requests because that format has no prompt roles to prepend or append. ## Plugin Attributes diff --git a/docs/en/latest/plugins/ai-prompt-guard.md b/docs/en/latest/plugins/ai-prompt-guard.md index 44546510ad71..7d57b745c5d8 100644 --- a/docs/en/latest/plugins/ai-prompt-guard.md +++ b/docs/en/latest/plugins/ai-prompt-guard.md @@ -40,7 +40,7 @@ The `ai-prompt-guard` Plugin safeguards your LLM endpoints by inspecting and val When both `allow_patterns` and `deny_patterns` are configured, the Plugin first ensures that at least one `allow_patterns` is matched. If none match, the request is rejected. If an allowed pattern is matched, it then checks for any occurrences of denied patterns. -The Plugin supports Chat Completions and Responses API requests. For Responses, it checks user content in `input` and, when `match_all_roles` is `true`, system content in `instructions`. For Embeddings, it checks `input` when it is a string; an array of input strings is not inspected. +The Plugin checks Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests using each protocol's native content structure. For Responses, it checks user content in `input` and, when `match_all_roles` is `true`, system content in `instructions`. For Embeddings, it checks `input` when it is a string; an array of input strings is not inspected. ## Plugin Attributes diff --git a/docs/en/latest/plugins/ai-proxy-multi.md b/docs/en/latest/plugins/ai-proxy-multi.md index 7e3fe242b57f..c1ace1c4183e 100644 --- a/docs/en/latest/plugins/ai-proxy-multi.md +++ b/docs/en/latest/plugins/ai-proxy-multi.md @@ -44,17 +44,19 @@ In addition, the Plugin also supports logging LLM request information in the acc ## Request Format -### OpenAI Request Protocol Detection +### Request Protocol Detection -For OpenAI and OpenAI-compatible requests, the Plugin detects the client protocol before selecting the corresponding upstream endpoint: +The Plugin detects the client request protocol before selecting a compatible upstream endpoint. It checks the following rules in order: | Client protocol | Detection | Route URI | | --- | --- | --- | -| Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | -| Responses API | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | -| Embeddings | The request body contains `input`, and neither the Responses API nor Chat Completions rule matched. | Any URI matched by the Route. | +| Bedrock Converse | The request body contains a `messages` array and the request URI ends in `/converse`. | The URI can have a custom prefix, but it must keep the `/converse` suffix. | +| Anthropic Messages | The request body is a JSON object and the request URI ends in `/v1/messages`. | The URI can have a custom prefix, but it must keep the `/v1/messages` suffix. | +| OpenAI Responses | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | +| OpenAI Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | +| OpenAI Embeddings | The request body contains `input`, and no earlier rule matched. | Any URI matched by the Route. | -The Plugin checks the rules in table order. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. +The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no rule matches, the Plugin treats the request as passthrough. ### Chat Completions Request Format diff --git a/docs/en/latest/plugins/ai-proxy.md b/docs/en/latest/plugins/ai-proxy.md index 211318564893..330cef1ddb10 100644 --- a/docs/en/latest/plugins/ai-proxy.md +++ b/docs/en/latest/plugins/ai-proxy.md @@ -44,17 +44,19 @@ In addition, the Plugin also supports logging LLM request information in the acc ## Request Format -### OpenAI Request Protocol Detection +### Request Protocol Detection -For OpenAI and OpenAI-compatible requests, the Plugin detects the client protocol before selecting the corresponding upstream endpoint: +The Plugin detects the client request protocol before selecting a compatible upstream endpoint. It checks the following rules in order: | Client protocol | Detection | Route URI | | --- | --- | --- | -| Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | -| Responses API | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | -| Embeddings | The request body contains `input`, and neither the Responses API nor Chat Completions rule matched. | Any URI matched by the Route. | +| Bedrock Converse | The request body contains a `messages` array and the request URI ends in `/converse`. | The URI can have a custom prefix, but it must keep the `/converse` suffix. | +| Anthropic Messages | The request body is a JSON object and the request URI ends in `/v1/messages`. | The URI can have a custom prefix, but it must keep the `/v1/messages` suffix. | +| OpenAI Responses | The request body contains `input` and the request URI ends in `/v1/responses`. | The URI can have a custom prefix, but it must keep the `/v1/responses` suffix. | +| OpenAI Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | +| OpenAI Embeddings | The request body contains `input`, and no earlier rule matched. | Any URI matched by the Route. | -The Plugin checks the rules in table order. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. +The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no rule matches, the Plugin treats the request as passthrough. ### Chat Completions Request Format diff --git a/docs/en/latest/plugins/ai-rag.md b/docs/en/latest/plugins/ai-rag.md index 7e351ffe7a99..b9f67c70f214 100644 --- a/docs/en/latest/plugins/ai-rag.md +++ b/docs/en/latest/plugins/ai-rag.md @@ -42,7 +42,7 @@ The `ai-rag` Plugin provides Retrieval-Augmented Generation (RAG) capabilities w The Plugin supports using [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) and [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) services for generating embeddings and performing vector search. PRs for introducing support for other service providers are welcomed. -The Plugin enriches Chat Completions and Responses API requests. For Responses, it appends retrieved context to `input`. It does not enrich direct Embeddings API requests. The nested `ai_rag.embeddings` object configures the embedding input used internally for retrieval and is distinct from a top-level Embeddings request. +The Plugin enriches Chat Completions, Responses API, Anthropic Messages, and Bedrock Converse requests using each protocol's native prompt structure. For Responses, it appends retrieved context to `input`. It does not enrich direct Embeddings API requests. The nested `ai_rag.embeddings` object configures the embedding input used internally for retrieval and is distinct from a top-level Embeddings request. ## Plugin Attributes diff --git a/docs/en/latest/plugins/ai-request-rewrite.md b/docs/en/latest/plugins/ai-request-rewrite.md index b689d7091abe..fa437b512b3b 100644 --- a/docs/en/latest/plugins/ai-request-rewrite.md +++ b/docs/en/latest/plugins/ai-request-rewrite.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-request-rewrite` Plugin processes client requests by forwarding them to LLM services for transformation before relaying them to Upstream services. This enables LLM-powered modifications such as data redaction, content enrichment, or reformatting. The Plugin supports integration with OpenAI, DeepSeek, Gemini, Vertex AI, Anthropic, OpenRouter, and other OpenAI-compatible APIs. -The LLM call used for rewriting is separate from the client's request format. With the `openai` provider, the Plugin builds a non-streaming Chat Completions request from the configured prompt and the original request body. It does not use the Chat Completions, Responses API, or Embeddings detection rules of [`ai-proxy`](./ai-proxy.md#openai-request-protocol-detection). +The LLM call used for rewriting is separate from the client's request format. With the `openai` provider, the Plugin builds a non-streaming Chat Completions request from the configured prompt and the original request body. It does not use [`ai-proxy` request protocol detection](./ai-proxy.md#request-protocol-detection). ## Plugin Attributes diff --git a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md index 9e5dc48aa079..88b4a03e7c9a 100644 --- a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md @@ -45,7 +45,7 @@ import TabItem from '@theme/TabItem'; `ai-aliyun-content-moderation` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件配合使用以代理 LLM 请求。 -该插件支持 Chat Completions、Responses API 和 Embeddings 请求。对于 Responses,它会审核 `input`,并以 Responses API 格式返回被拦截的非流式或流式请求。对于 Embeddings,它会审核 `input` 中的字符串或字符串数组,并以 OpenAI 风格的错误响应返回被拒绝的请求。 +该插件使用每种协议的原生内容结构审核 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会审核 `input`,并以 Responses API 格式返回被拦截的非流式或流式请求。对于 Embeddings,它会审核 `input` 中的字符串或字符串数组,并以 OpenAI 风格的错误响应返回被拒绝的请求。 ## 属性 diff --git a/docs/zh/latest/plugins/ai-aws-content-moderation.md b/docs/zh/latest/plugins/ai-aws-content-moderation.md index 27a83cd28664..0c0c7015545e 100644 --- a/docs/zh/latest/plugins/ai-aws-content-moderation.md +++ b/docs/zh/latest/plugins/ai-aws-content-moderation.md @@ -40,7 +40,7 @@ import TabItem from '@theme/TabItem'; `ai-aws-content-moderation` 插件集成了 [AWS Comprehend](https://aws.amazon.com/comprehend/),用于在代理请求到 LLM 时检查请求内容中的有害内容,例如亵渎、仇恨言论、侮辱、骚扰、暴力等,如果评估结果超过配置的阈值则拒绝请求。 -该插件支持检测到的 Chat Completions、Responses API 和 Embeddings 请求。它会分别从 `messages`、Responses 的 `instructions` 和 `input`,或 Embeddings 的 `input` 中提取文本,并仅审核解码后的文本,而不是原始请求体。被拒绝的请求使用检测到的协议对应的响应格式。 +该插件支持 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。它会使用检测到的协议的原生内容结构提取文本,并仅审核解码后的文本,而不是原始请求体。被拒绝的请求使用检测到的协议对应的响应格式。 `ai-aws-content-moderation` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件一起使用,以代理 LLM 请求。 diff --git a/docs/zh/latest/plugins/ai-cache.md b/docs/zh/latest/plugins/ai-cache.md index 3437e77ac22f..f46524de2f94 100644 --- a/docs/zh/latest/plugins/ai-cache.md +++ b/docs/zh/latest/plugins/ai-cache.md @@ -45,7 +45,7 @@ import TabItem from '@theme/TabItem'; - **精确缓存(L1):** 对有效提示词计算 SHA-256 指纹并用作 Redis 键。完全相同的提示词始终命中同一条缓存条目。 - **语义缓存(L2):** 当 L1 未命中时,将提示词向量化,并通过最近邻搜索检索相似度在阈值以上的历史响应。L2 默认关闭;在 `layers` 中加入 `"semantic"` 即可启用。 -精确缓存支持检测到的 Chat Completions、Responses API 和 Embeddings 请求,并将不同协议存储在独立的缓存条目中。语义缓存仅适用于 Chat Completions 请求;Responses 和 Embeddings 请求可以使用精确缓存层,但会绕过语义缓存层。 +精确缓存支持 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求,并将不同协议存储在独立的缓存条目中。语义缓存仅适用于 Chat Completions 请求;其他协议可以使用精确缓存层,但会绕过语义缓存层。 `ai-cache` 插件必须与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件一起使用。 diff --git a/docs/zh/latest/plugins/ai-lakera-guard.md b/docs/zh/latest/plugins/ai-lakera-guard.md index 75c5dc05af3f..2c07be2a91c8 100644 --- a/docs/zh/latest/plugins/ai-lakera-guard.md +++ b/docs/zh/latest/plugins/ai-lakera-guard.md @@ -45,7 +45,7 @@ import TabItem from '@theme/TabItem'; `ai-lakera-guard` 插件应与 [`ai-proxy`](./ai-proxy.md) 或 [`ai-proxy-multi`](./ai-proxy-multi.md) 插件配合使用以代理 LLM 请求。它依赖 `ai-proxy` 填充的上下文,以协议感知的方式提取请求内容。 -请求扫描支持 Chat Completions、Responses API 和 Embeddings 请求。对于 Responses,插件会将 `instructions` 和 `input` 中的文本转换为对话消息。对于 Embeddings,插件会扫描 `input` 中的文本。响应扫描适用于返回生成文本的协议,包括 Chat Completions 和 Responses;Embeddings 响应包含向量而不是文本。 +请求扫描支持 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,插件会将 `instructions` 和 `input` 中的文本转换为对话消息。对于 Embeddings,插件会扫描 `input` 中的文本。响应扫描适用于返回生成文本的协议;Embeddings 响应包含向量而不是文本。 未经过 `ai-proxy`/`ai-proxy-multi` 的请求(例如插件绑定在 Consumer 或 Service 级别时的普通 HTTP 流量)无法被检查。默认情况下,此类请求会被直接放行而不做检查;该行为可通过 `fail_mode` 配置。 diff --git a/docs/zh/latest/plugins/ai-prompt-decorator.md b/docs/zh/latest/plugins/ai-prompt-decorator.md index 40c70d12ef80..6754f1e7134b 100644 --- a/docs/zh/latest/plugins/ai-prompt-decorator.md +++ b/docs/zh/latest/plugins/ai-prompt-decorator.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; `ai-prompt-decorator` 插件通过在用户输入提示前后添加预设提示来修改用户输入,以在内容生成中设定上下文。这种做法有助于模型在交互过程中按照预期的指导方针运行。 -该插件支持 Chat Completions 和 Responses API 请求。对于 Responses,`prepend` 中的内容会添加到 `instructions`,`append` 中的内容会添加到 `input`。该插件不会修改 Embeddings 请求,因为该格式没有可前置或追加的提示角色。 +该插件使用每种协议的原生提示结构修饰 Chat Completions、Responses API、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,`prepend` 中的内容会添加到 `instructions`,`append` 中的内容会添加到 `input`。该插件不会修改 Embeddings 请求,因为该格式没有可前置或追加的提示角色。 ## 插件属性 diff --git a/docs/zh/latest/plugins/ai-prompt-guard.md b/docs/zh/latest/plugins/ai-prompt-guard.md index 32f24abdd7d9..86c432964785 100644 --- a/docs/zh/latest/plugins/ai-prompt-guard.md +++ b/docs/zh/latest/plugins/ai-prompt-guard.md @@ -40,7 +40,7 @@ import TabItem from '@theme/TabItem'; 当同时配置了 `allow_patterns` 和 `deny_patterns` 时,插件首先确保至少匹配一个 `allow_patterns`。如果没有匹配,请求将被拒绝。如果匹配了允许的模式,它会继续检查是否存在任何拒绝模式的匹配。 -该插件支持 Chat Completions 和 Responses API 请求。对于 Responses,它会检查 `input` 中的用户内容;当 `match_all_roles` 为 `true` 时,还会检查 `instructions` 中的系统内容。对于 Embeddings,它会在 `input` 为字符串时进行检查;字符串数组形式的输入不会被检查。 +该插件使用每种协议的原生内容结构检查 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会检查 `input` 中的用户内容;当 `match_all_roles` 为 `true` 时,还会检查 `instructions` 中的系统内容。对于 Embeddings,它会在 `input` 为字符串时进行检查;字符串数组形式的输入不会被检查。 ## 插件属性 diff --git a/docs/zh/latest/plugins/ai-proxy-multi.md b/docs/zh/latest/plugins/ai-proxy-multi.md index 9a95a1563307..df1d7ae98c08 100644 --- a/docs/zh/latest/plugins/ai-proxy-multi.md +++ b/docs/zh/latest/plugins/ai-proxy-multi.md @@ -44,17 +44,19 @@ import TabItem from '@theme/TabItem'; ## 请求格式 -### OpenAI 请求协议检测 +### 请求协议检测 -对于 OpenAI 和 OpenAI 兼容请求,插件会先检测客户端协议,再选择对应的上游端点: +插件会先检测客户端请求协议,再选择兼容的上游端点。插件按以下顺序检查规则: | 客户端协议 | 检测条件 | 路由 URI | | --- | --- | --- | -| Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | -| Responses API | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | -| Embeddings | 请求体包含 `input`,并且 Responses API 和 Chat Completions 规则均未匹配。 | 路由匹配的任意 URI。 | +| Bedrock Converse | 请求体包含 `messages` 数组,并且请求 URI 以 `/converse` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/converse` 后缀。 | +| Anthropic Messages | 请求体为 JSON 对象,并且请求 URI 以 `/v1/messages` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/messages` 后缀。 | +| OpenAI Responses | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | +| OpenAI Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | +| OpenAI Embeddings | 请求体包含 `input`,并且之前的规则均未匹配。 | 路由匹配的任意 URI。 | -插件按表中顺序检查规则。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。 +基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果没有规则匹配,插件会将请求视为透传请求。 ### Chat Completions 请求格式 diff --git a/docs/zh/latest/plugins/ai-proxy.md b/docs/zh/latest/plugins/ai-proxy.md index 45da3736fad4..eaf344714b43 100644 --- a/docs/zh/latest/plugins/ai-proxy.md +++ b/docs/zh/latest/plugins/ai-proxy.md @@ -44,17 +44,19 @@ import TabItem from '@theme/TabItem'; ## 请求格式 -### OpenAI 请求协议检测 +### 请求协议检测 -对于 OpenAI 和 OpenAI 兼容请求,插件会先检测客户端协议,再选择对应的上游端点: +插件会先检测客户端请求协议,再选择兼容的上游端点。插件按以下顺序检查规则: | 客户端协议 | 检测条件 | 路由 URI | | --- | --- | --- | -| Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | -| Responses API | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | -| Embeddings | 请求体包含 `input`,并且 Responses API 和 Chat Completions 规则均未匹配。 | 路由匹配的任意 URI。 | +| Bedrock Converse | 请求体包含 `messages` 数组,并且请求 URI 以 `/converse` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/converse` 后缀。 | +| Anthropic Messages | 请求体为 JSON 对象,并且请求 URI 以 `/v1/messages` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/messages` 后缀。 | +| OpenAI Responses | 请求体包含 `input`,并且请求 URI 以 `/v1/responses` 结尾。 | URI 可以包含自定义前缀,但必须保留 `/v1/responses` 后缀。 | +| OpenAI Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | +| OpenAI Embeddings | 请求体包含 `input`,并且之前的规则均未匹配。 | 路由匹配的任意 URI。 | -插件按表中顺序检查规则。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。 +基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果没有规则匹配,插件会将请求视为透传请求。 ### Chat Completions 请求格式 diff --git a/docs/zh/latest/plugins/ai-rag.md b/docs/zh/latest/plugins/ai-rag.md index 60195d2c27d1..8593f3077ada 100644 --- a/docs/zh/latest/plugins/ai-rag.md +++ b/docs/zh/latest/plugins/ai-rag.md @@ -42,7 +42,7 @@ import TabItem from '@theme/TabItem'; 该插件支持使用 [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) 和 [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) 服务来生成嵌入和执行向量搜索。欢迎提交 PR 以引入对其他服务提供商的支持。 -该插件会增强 Chat Completions 和 Responses API 请求。对于 Responses,它会将检索到的上下文追加到 `input`。它不会增强直接的 Embeddings API 请求。嵌套的 `ai_rag.embeddings` 对象用于配置检索过程中内部使用的嵌入输入,与顶层 Embeddings 请求不同。 +该插件使用每种协议的原生提示结构增强 Chat Completions、Responses API、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会将检索到的上下文追加到 `input`。它不会增强直接的 Embeddings API 请求。嵌套的 `ai_rag.embeddings` 对象用于配置检索过程中内部使用的嵌入输入,与顶层 Embeddings 请求不同。 ## 插件属性 diff --git a/docs/zh/latest/plugins/ai-request-rewrite.md b/docs/zh/latest/plugins/ai-request-rewrite.md index f9eb89c92b95..415b160b155c 100644 --- a/docs/zh/latest/plugins/ai-request-rewrite.md +++ b/docs/zh/latest/plugins/ai-request-rewrite.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; `ai-request-rewrite` 插件在将客户端请求转发到上游服务之前,先将请求发送到 LLM 服务进行转换处理。这使得 LLM 能够对请求进行数据脱敏、内容增强或格式转换等修改。该插件支持集成 OpenAI、DeepSeek、Gemini、Vertex AI、Anthropic、OpenRouter 以及其他 OpenAI 兼容的 API。 -用于重写的 LLM 调用与客户端请求格式相互独立。使用 `openai` 提供商时,插件会根据配置的提示词和原始请求体构建非流式 Chat Completions 请求。它不会使用 [`ai-proxy`](./ai-proxy.md#openai-请求协议检测) 的 Chat Completions、Responses API 或 Embeddings 检测规则。 +用于重写的 LLM 调用与客户端请求格式相互独立。使用 `openai` 提供商时,插件会根据配置的提示词和原始请求体构建非流式 Chat Completions 请求。它不会使用 [`ai-proxy` 请求协议检测](./ai-proxy.md#请求协议检测)。 ## 插件属性 From 369ed4f9805509be540145ca40709bf61704a08c Mon Sep 17 00:00:00 2001 From: kayx23 Date: Fri, 24 Jul 2026 16:52:03 +0800 Subject: [PATCH 5/5] docs(ai): sync final protocol guidance --- .../plugins/ai-aliyun-content-moderation.md | 8 ++ docs/en/latest/plugins/ai-prompt-decorator.md | 8 ++ docs/en/latest/plugins/ai-prompt-guard.md | 8 ++ docs/en/latest/plugins/ai-prompt-template.md | 119 +++++++++++++++++- docs/en/latest/plugins/ai-proxy-multi.md | 4 +- docs/en/latest/plugins/ai-proxy.md | 4 +- docs/en/latest/plugins/ai-rag.md | 8 ++ docs/en/latest/plugins/ai-request-rewrite.md | 2 +- .../plugins/ai-aliyun-content-moderation.md | 8 ++ docs/zh/latest/plugins/ai-prompt-decorator.md | 8 ++ docs/zh/latest/plugins/ai-prompt-guard.md | 8 ++ docs/zh/latest/plugins/ai-prompt-template.md | 119 +++++++++++++++++- docs/zh/latest/plugins/ai-proxy-multi.md | 4 +- docs/zh/latest/plugins/ai-proxy.md | 4 +- docs/zh/latest/plugins/ai-rag.md | 8 ++ docs/zh/latest/plugins/ai-request-rewrite.md | 2 +- 16 files changed, 308 insertions(+), 14 deletions(-) diff --git a/docs/en/latest/plugins/ai-aliyun-content-moderation.md b/docs/en/latest/plugins/ai-aliyun-content-moderation.md index 24c30852b445..30528fb002e6 100644 --- a/docs/en/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/en/latest/plugins/ai-aliyun-content-moderation.md @@ -47,6 +47,14 @@ The `ai-aliyun-content-moderation` Plugin should be used with either [`ai-proxy` The Plugin moderates Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests using each protocol's native content structure. For Responses, it moderates `input` and returns blocked non-streaming or streaming requests in Responses API format. For Embeddings, it moderates a string or an array of strings in `input` and returns rejected requests as OpenAI-style errors. +APISIX checks URI-specific formats before body-only formats: + +- Bedrock Converse requires a URI ending in `/converse` and a `messages` array. +- Anthropic Messages requires a URI ending in `/v1/messages`. +- Responses API requires a URI ending in `/v1/responses` and an `input` field. +- Chat Completions uses a `messages` array. +- Embeddings uses `input` after the earlier rules do not match. + ## Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-prompt-decorator.md b/docs/en/latest/plugins/ai-prompt-decorator.md index fa3d938ce3c2..b876dcd8e430 100644 --- a/docs/en/latest/plugins/ai-prompt-decorator.md +++ b/docs/en/latest/plugins/ai-prompt-decorator.md @@ -40,6 +40,14 @@ The `ai-prompt-decorator` Plugin modifies user input prompts by prefixing and ap The Plugin decorates Chat Completions, Responses API, Anthropic Messages, and Bedrock Converse requests using each protocol's native prompt structure. For Responses, content from `prepend` is added to `instructions`, and content from `append` is added to `input`. The Plugin does not modify Embeddings requests because that format has no prompt roles to prepend or append. +APISIX checks URI-specific formats before body-only formats: + +- Bedrock Converse requires a URI ending in `/converse` and a `messages` array. +- Anthropic Messages requires a URI ending in `/v1/messages`. +- Responses API requires a URI ending in `/v1/responses` and an `input` field. +- Chat Completions uses a `messages` array. +- Embeddings uses `input` after the earlier rules do not match. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-prompt-guard.md b/docs/en/latest/plugins/ai-prompt-guard.md index 1806211db42d..4ff8ad880da8 100644 --- a/docs/en/latest/plugins/ai-prompt-guard.md +++ b/docs/en/latest/plugins/ai-prompt-guard.md @@ -42,6 +42,14 @@ When both `allow_patterns` and `deny_patterns` are configured, the Plugin first The Plugin checks Chat Completions, Responses API, Embeddings, Anthropic Messages, and Bedrock Converse requests using each protocol's native content structure. For Responses, it checks user content in `input` and, when `match_all_roles` is `true`, system content in `instructions`. For Embeddings, it checks `input` when it is a string; an array of input strings is not inspected. +APISIX checks URI-specific formats before body-only formats: + +- Bedrock Converse requires a URI ending in `/converse` and a `messages` array. +- Anthropic Messages requires a URI ending in `/v1/messages`. +- Responses API requires a URI ending in `/v1/responses` and an `input` field. +- Chat Completions uses a `messages` array. +- Embeddings uses `input` after the earlier rules do not match. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-prompt-template.md b/docs/en/latest/plugins/ai-prompt-template.md index 321df312153a..8d879b172104 100644 --- a/docs/en/latest/plugins/ai-prompt-template.md +++ b/docs/en/latest/plugins/ai-prompt-template.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-prompt-template` Plugin supports pre-configuring prompt templates that only accept user inputs in designated template variables, in a "fill in the blank" fashion. It simplifies access to LLM providers, such as OpenAI and Anthropic, by letting you define reusable prompt structures. -The Plugin generates a Chat Completions request containing a `messages` array. It does not generate Responses API or Embeddings requests. +The Plugin replaces the incoming request body with the selected rendered JSON template. A template can use any request format supported by the downstream Plugin, including Chat Completions, Responses API, and Embeddings. Use [`ai-prompt-decorator`](./ai-prompt-decorator.md) instead when you need to add content to an existing request without replacing its body. ## Plugin Attributes @@ -47,7 +47,7 @@ The Plugin generates a Chat Completions request containing a `messages` array. I | `max_req_body_size` | integer | False | 67108864 | >= 1 | Maximum request body size in bytes buffered into memory. Requests with a larger body are rejected. | | `templates` | array | True | | | An array of template objects. | | `templates.name` | string | True | | | Name of the template. When requesting the Route, the request should include the template name that corresponds to the configured template. | -| `templates.template` | object | True | | | Template specification. | +| `templates.template` | object | True | | | JSON request body template. It can contain any fields accepted by the downstream Plugin. | | `templates.template.model` | string | False | | | Name of the LLM model, such as `gpt-4` or `gpt-3.5`. See your LLM provider API documentation for more available models. | | `templates.template.messages` | array[object] | False | | | Template message specification. | | `templates.template.messages.role` | string | True | | [`system`, `user`, `assistant`] | Role of the message. | @@ -603,3 +603,118 @@ You should receive a response similar to the following: ... } ``` + +### Configure a Responses API Web Search Template + +The following example configures a Responses API template that searches only the official Apache APISIX website. The Route URI can have a custom prefix, but it must end in `/v1/responses` so that `ai-proxy` distinguishes the rendered body from an Embeddings request. + +Create a Route with the `ai-proxy` and `ai-prompt-template` Plugins: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes/ai-prompt-template-responses" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "uri": "/template/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy": { + "provider": "openai", + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + } + }, + "ai-prompt-template": { + "templates": [ + { + "name": "Search APISIX documentation", + "template": { + "model": "gpt-4.1", + "tools": [ + { + "type": "web_search", + "filters": { + "allowed_domains": ["apisix.apache.org"] + } + } + ], + "tool_choice": "required", + "input": "Search the official Apache APISIX website for {{topic}} and summarize the result in one sentence." + } + } + ] + } + } + }' +``` + +Send the template name and a value for `topic` to the Route: + +```shell +curl "http://127.0.0.1:9080/template/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "template_name": "Search APISIX documentation", + "topic": "what APISIX is" + }' +``` + +Before `ai-proxy` processes the request, `ai-prompt-template` replaces the incoming body with the rendered template: + +```json +{ + "model": "gpt-4.1", + "tools": [ + { + "type": "web_search", + "filters": { + "allowed_domains": ["apisix.apache.org"] + } + } + ], + "tool_choice": "required", + "input": "Search the official Apache APISIX website for what APISIX is and summarize the result in one sentence." +} +``` + +APISIX identifies the rendered request as Responses API from its `input` field and the Route URI suffix. You should receive an HTTP `200` response whose `output` contains a completed web search call and a message with URL citations: + +```json +{ + "status": "completed", + "output": [ + { + "type": "web_search_call", + "status": "completed", + "action": { + "type": "search", + "queries": [ + "site:apisix.apache.org what is Apache APISIX" + ], + "query": "site:apisix.apache.org what is Apache APISIX" + } + }, + { + "type": "message", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Apache APISIX is an open-source, high-performance API and AI gateway for managing traffic at scale through dynamic routing, load balancing, authentication, observability, rate limiting, and over 100 plugins. ([apisix.apache.org](https://apisix.apache.org/?utm_source=openai))", + "annotations": [ + { + "type": "url_citation", + "start_index": 208, + "end_index": 275, + "title": "Apache APISIX - Open Source API Gateway & AI Gateway", + "url": "https://apisix.apache.org/?utm_source=openai" + } + ] + } + ] + } + ] +} +``` diff --git a/docs/en/latest/plugins/ai-proxy-multi.md b/docs/en/latest/plugins/ai-proxy-multi.md index 153ffbea2859..2ddf931ddc84 100644 --- a/docs/en/latest/plugins/ai-proxy-multi.md +++ b/docs/en/latest/plugins/ai-proxy-multi.md @@ -56,7 +56,7 @@ The Plugin detects the client request protocol before selecting a compatible ups | OpenAI Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | | OpenAI Embeddings | The request body contains `input`, and no earlier rule matched. | Any URI matched by the Route. | -The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no rule matches, the Plugin treats the request as passthrough. +The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no earlier rule matches, the Plugin treats a non-empty JSON object as passthrough. Empty or invalid request bodies are rejected. ### Chat Completions Request Format @@ -92,7 +92,7 @@ When an instance's `provider` is set to `bedrock`, the Plugin expects requests i | balancer.key | string | False | | | Used when `type` is `chash`. When `hash_on` is set to `header` or `cookie`, `key` is required. When `hash_on` is set to `consumer`, `key` is not required as the consumer name will be used as the key automatically. | | instances | array[object] | True | | | LLM instance configurations. | | instances.name | string | True | | | Name of the LLM service instance. | -| instances.provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding endpoint on `api.openai.com`. When set to `deepseek`, the Plugin will proxy the request to `api.deepseek.com`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `api.aimlapi.com` by default. When set to `anthropic`, the Plugin will proxy the request to `api.anthropic.com` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `openrouter.ai` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `generativelanguage.googleapis.com` by default. When set to `vertex-ai`, the Plugin will proxy the request to `aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin proxies the request to Amazon Bedrock's Converse API at `bedrock-runtime.{region}.amazonaws.com` and signs the request with AWS SigV4. Requires `provider_conf.region` and `auth.aws`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | +| instances.provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding endpoint on `api.openai.com`. When set to `deepseek`, the Plugin will proxy the request to `api.deepseek.com`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `api.aimlapi.com` by default. When set to `anthropic`, the Plugin sends detected Chat Completions requests to `/v1/chat/completions` and native Anthropic Messages requests to `/v1/messages` on `api.anthropic.com`. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `openrouter.ai` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `generativelanguage.googleapis.com` by default. When set to `vertex-ai`, the Plugin will proxy the request to `aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin proxies the request to Amazon Bedrock's Converse API at `bedrock-runtime.{region}.amazonaws.com` and signs the request with AWS SigV4. Requires `provider_conf.region` and `auth.aws`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | instances.provider_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. Required when `provider` is set to `bedrock`. | | instances.provider_conf.project_id | string | True | | | Google Cloud Project ID. | | instances.provider_conf.region | string | True (depending on provider) | | minLength = 1 (for Bedrock) | When `provider` is `vertex-ai`, this is the Google Cloud Region. When `provider` is `bedrock`, this is the AWS region used to construct the Bedrock endpoint and to sign the request with SigV4 (required, must be non-empty). | diff --git a/docs/en/latest/plugins/ai-proxy.md b/docs/en/latest/plugins/ai-proxy.md index 441ef4ada63d..1736122686bc 100644 --- a/docs/en/latest/plugins/ai-proxy.md +++ b/docs/en/latest/plugins/ai-proxy.md @@ -56,7 +56,7 @@ The Plugin detects the client request protocol before selecting a compatible ups | OpenAI Chat Completions | The request body contains a `messages` array. | Any URI matched by the Route. | | OpenAI Embeddings | The request body contains `input`, and no earlier rule matched. | Any URI matched by the Route. | -The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no rule matches, the Plugin treats the request as passthrough. +The URI-specific rules run before the body-only rules. This prevents Bedrock Converse and Anthropic Messages requests containing `messages` from being identified as Chat Completions. Responses and Embeddings requests both use `input`, so a request containing `input` but not `messages` is identified as Embeddings unless its URI ends in `/v1/responses`. If no earlier rule matches, the Plugin treats a non-empty JSON object as passthrough. Empty or invalid request bodies are rejected. ### Chat Completions Request Format @@ -83,7 +83,7 @@ When `provider` is set to `bedrock`, the Plugin expects requests in the [Bedrock | Name | Type | Required | Default | Valid values | Description | |--------------------|--------|----------|---------|------------------------------------------|-------------| -| provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding OpenAI endpoint. When set to `deepseek`, the Plugin will proxy the request to `https://api.deepseek.com/chat/completions`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions` by default. When set to `anthropic`, the Plugin will proxy the request to `https://api.anthropic.com/v1/chat/completions` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://openrouter.ai/api/v1/chat/completions` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` by default. When set to `vertex-ai`, the Plugin will proxy the request to `https://aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin will proxy the request to the AWS Bedrock Converse API (`https://bedrock-runtime..amazonaws.com`) and signs the request with AWS SigV4. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | +| provider | string | True | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM service provider. When set to `openai`, the Plugin sends detected Chat Completions, Responses, and Embeddings requests to the corresponding OpenAI endpoint. When set to `deepseek`, the Plugin will proxy the request to `https://api.deepseek.com/chat/completions`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions` by default. When set to `anthropic`, the Plugin sends detected Chat Completions requests to `https://api.anthropic.com/v1/chat/completions` and native Anthropic Messages requests to `https://api.anthropic.com/v1/messages`. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://openrouter.ai/api/v1/chat/completions` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` by default. When set to `vertex-ai`, the Plugin will proxy the request to `https://aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `bedrock`, the Plugin will proxy the request to the AWS Bedrock Converse API (`https://bedrock-runtime..amazonaws.com`) and signs the request with AWS SigV4. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | provider_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. Required when `provider` is set to `bedrock`. | | provider_conf.project_id | string | True | | | Google Cloud Project ID. | | provider_conf.region | string | True (depending on provider) | | minLength = 1 (for Bedrock) | When `provider` is `vertex-ai`, this is the Google Cloud Region. When `provider` is `bedrock`, this is the AWS region used to construct the Bedrock endpoint and to sign the request with SigV4 (required, must be non-empty). | diff --git a/docs/en/latest/plugins/ai-rag.md b/docs/en/latest/plugins/ai-rag.md index b9f67c70f214..9ea00f1e60ea 100644 --- a/docs/en/latest/plugins/ai-rag.md +++ b/docs/en/latest/plugins/ai-rag.md @@ -44,6 +44,14 @@ The Plugin supports using [Azure OpenAI](https://azure.microsoft.com/en-us/produ The Plugin enriches Chat Completions, Responses API, Anthropic Messages, and Bedrock Converse requests using each protocol's native prompt structure. For Responses, it appends retrieved context to `input`. It does not enrich direct Embeddings API requests. The nested `ai_rag.embeddings` object configures the embedding input used internally for retrieval and is distinct from a top-level Embeddings request. +APISIX checks URI-specific formats before body-only formats: + +- Bedrock Converse requires a URI ending in `/converse` and a `messages` array. +- Anthropic Messages requires a URI ending in `/v1/messages`. +- Responses API requires a URI ending in `/v1/responses` and an `input` field. +- Chat Completions uses a `messages` array. +- Embeddings uses `input` after the earlier rules do not match. + ## Plugin Attributes | Name | Type | Required | Default | Valid values | Description | diff --git a/docs/en/latest/plugins/ai-request-rewrite.md b/docs/en/latest/plugins/ai-request-rewrite.md index c54b3df3f88e..802c4e66f4ce 100644 --- a/docs/en/latest/plugins/ai-request-rewrite.md +++ b/docs/en/latest/plugins/ai-request-rewrite.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; The `ai-request-rewrite` Plugin processes client requests by forwarding them to LLM services for transformation before relaying them to Upstream services. This enables LLM-powered modifications such as data redaction, content enrichment, or reformatting. The Plugin supports integration with OpenAI, DeepSeek, Gemini, Vertex AI, Anthropic, OpenRouter, and other OpenAI-compatible APIs. -The LLM call used for rewriting is separate from the client's request format. With the `openai` provider, the Plugin builds a non-streaming Chat Completions request from the configured prompt and the original request body. It does not use [`ai-proxy` request protocol detection](./ai-proxy.md#request-protocol-detection). +The LLM call used for rewriting is separate from the client's request format. With the `openai` provider, the Plugin builds a non-streaming Chat Completions request from the configured prompt and the original request body. It does not classify or proxy the client's request as an AI protocol. ## Plugin Attributes diff --git a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md index 88b4a03e7c9a..dbdafd670853 100644 --- a/docs/zh/latest/plugins/ai-aliyun-content-moderation.md +++ b/docs/zh/latest/plugins/ai-aliyun-content-moderation.md @@ -47,6 +47,14 @@ import TabItem from '@theme/TabItem'; 该插件使用每种协议的原生内容结构审核 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会审核 `input`,并以 Responses API 格式返回被拦截的非流式或流式请求。对于 Embeddings,它会审核 `input` 中的字符串或字符串数组,并以 OpenAI 风格的错误响应返回被拒绝的请求。 +APISIX 会先检查基于 URI 的格式,再检查仅基于请求体的格式: + +- Bedrock Converse 要求 URI 以 `/converse` 结尾且请求体包含 `messages` 数组。 +- Anthropic Messages 要求 URI 以 `/v1/messages` 结尾。 +- Responses API 要求 URI 以 `/v1/responses` 结尾且请求体包含 `input`。 +- Chat Completions 使用 `messages` 数组。 +- 之前的规则均未匹配时,包含 `input` 的请求会被识别为 Embeddings。 + ## 属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-prompt-decorator.md b/docs/zh/latest/plugins/ai-prompt-decorator.md index 6754f1e7134b..bc35491f65a3 100644 --- a/docs/zh/latest/plugins/ai-prompt-decorator.md +++ b/docs/zh/latest/plugins/ai-prompt-decorator.md @@ -40,6 +40,14 @@ import TabItem from '@theme/TabItem'; 该插件使用每种协议的原生提示结构修饰 Chat Completions、Responses API、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,`prepend` 中的内容会添加到 `instructions`,`append` 中的内容会添加到 `input`。该插件不会修改 Embeddings 请求,因为该格式没有可前置或追加的提示角色。 +APISIX 会先检查基于 URI 的格式,再检查仅基于请求体的格式: + +- Bedrock Converse 要求 URI 以 `/converse` 结尾且请求体包含 `messages` 数组。 +- Anthropic Messages 要求 URI 以 `/v1/messages` 结尾。 +- Responses API 要求 URI 以 `/v1/responses` 结尾且请求体包含 `input`。 +- Chat Completions 使用 `messages` 数组。 +- 之前的规则均未匹配时,包含 `input` 的请求会被识别为 Embeddings。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-prompt-guard.md b/docs/zh/latest/plugins/ai-prompt-guard.md index 86c432964785..18443d1c50ad 100644 --- a/docs/zh/latest/plugins/ai-prompt-guard.md +++ b/docs/zh/latest/plugins/ai-prompt-guard.md @@ -42,6 +42,14 @@ import TabItem from '@theme/TabItem'; 该插件使用每种协议的原生内容结构检查 Chat Completions、Responses API、Embeddings、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会检查 `input` 中的用户内容;当 `match_all_roles` 为 `true` 时,还会检查 `instructions` 中的系统内容。对于 Embeddings,它会在 `input` 为字符串时进行检查;字符串数组形式的输入不会被检查。 +APISIX 会先检查基于 URI 的格式,再检查仅基于请求体的格式: + +- Bedrock Converse 要求 URI 以 `/converse` 结尾且请求体包含 `messages` 数组。 +- Anthropic Messages 要求 URI 以 `/v1/messages` 结尾。 +- Responses API 要求 URI 以 `/v1/responses` 结尾且请求体包含 `input`。 +- Chat Completions 使用 `messages` 数组。 +- 之前的规则均未匹配时,包含 `input` 的请求会被识别为 Embeddings。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-prompt-template.md b/docs/zh/latest/plugins/ai-prompt-template.md index 13247f2e4aae..a232cca8c155 100644 --- a/docs/zh/latest/plugins/ai-prompt-template.md +++ b/docs/zh/latest/plugins/ai-prompt-template.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; `ai-prompt-template` 插件支持预先配置提示词模板,这些模板仅接受用户在指定的模板变量中输入,采用"填空"的方式。它简化了对 OpenAI、Anthropic 等大语言模型提供商及其模型的访问,让你可以定义可复用的提示词结构。 -该插件会生成包含 `messages` 数组的 Chat Completions 请求。它不会生成 Responses API 或 Embeddings 请求。 +该插件会使用选定并渲染后的 JSON 模板替换传入的请求体。模板可以使用下游插件支持的任意请求格式,包括 Chat Completions、Responses API 和 Embeddings。如果需要在不替换请求体的情况下向现有请求添加内容,请改用 [`ai-prompt-decorator`](./ai-prompt-decorator.md)。 ## 插件属性 @@ -46,7 +46,7 @@ import TabItem from '@theme/TabItem'; | --- | --- | --- | --- | --- | --- | | `templates` | array | 是 | | | 模板对象数组。 | | `templates.name` | string | 是 | | | 模板的名称。在请求路由时,请求中应包含与所配置模板相对应的模板名称。 | -| `templates.template` | object | 是 | | | 模板规范。 | +| `templates.template` | object | 是 | | | JSON 请求体模板。它可以包含下游插件接受的任意字段。 | | `templates.template.model` | string | 否 | | | LLM 模型的名称,例如 `gpt-4` 或 `gpt-3.5`。更多可用模型请参阅 LLM 提供商的 API 文档。 | | `templates.template.messages` | array[object] | 否 | | | 模板消息规范。 | | `templates.template.messages.role` | string | 是 | | [`system`, `user`, `assistant`] | 消息的角色。 | @@ -602,3 +602,118 @@ curl "http://127.0.0.1:9080/openai-chat" -X POST \ ... } ``` + +### 配置 Responses API Web 搜索模板 + +以下示例配置一个仅搜索 Apache APISIX 官方网站的 Responses API 模板。路由 URI 可以使用自定义前缀,但必须以 `/v1/responses` 结尾,以便 `ai-proxy` 将渲染后的请求体与 Embeddings 请求区分开。 + +创建一个配置了 `ai-proxy` 和 `ai-prompt-template` 插件的路由: + +```shell +curl "http://127.0.0.1:9180/apisix/admin/routes/ai-prompt-template-responses" -X PUT \ + -H "X-API-KEY: ${admin_key}" \ + -d '{ + "uri": "/template/v1/responses", + "methods": ["POST"], + "plugins": { + "ai-proxy": { + "provider": "openai", + "auth": { + "header": { + "Authorization": "Bearer '"$OPENAI_API_KEY"'" + } + } + }, + "ai-prompt-template": { + "templates": [ + { + "name": "Search APISIX documentation", + "template": { + "model": "gpt-4.1", + "tools": [ + { + "type": "web_search", + "filters": { + "allowed_domains": ["apisix.apache.org"] + } + } + ], + "tool_choice": "required", + "input": "Search the official Apache APISIX website for {{topic}} and summarize the result in one sentence." + } + } + ] + } + } + }' +``` + +向路由发送模板名称和 `topic` 的值: + +```shell +curl "http://127.0.0.1:9080/template/v1/responses" -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "template_name": "Search APISIX documentation", + "topic": "what APISIX is" + }' +``` + +在 `ai-proxy` 处理请求之前,`ai-prompt-template` 会使用渲染后的模板替换传入的请求体: + +```json +{ + "model": "gpt-4.1", + "tools": [ + { + "type": "web_search", + "filters": { + "allowed_domains": ["apisix.apache.org"] + } + } + ], + "tool_choice": "required", + "input": "Search the official Apache APISIX website for what APISIX is and summarize the result in one sentence." +} +``` + +APISIX 根据渲染后请求中的 `input` 字段和路由 URI 后缀将其识别为 Responses API 请求。你应该会收到 HTTP `200` 响应,其中的 `output` 包含已完成的 Web 搜索调用和带有 URL 引用的消息: + +```json +{ + "status": "completed", + "output": [ + { + "type": "web_search_call", + "status": "completed", + "action": { + "type": "search", + "queries": [ + "site:apisix.apache.org what is Apache APISIX" + ], + "query": "site:apisix.apache.org what is Apache APISIX" + } + }, + { + "type": "message", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Apache APISIX is an open-source, high-performance API and AI gateway for managing traffic at scale through dynamic routing, load balancing, authentication, observability, rate limiting, and over 100 plugins. ([apisix.apache.org](https://apisix.apache.org/?utm_source=openai))", + "annotations": [ + { + "type": "url_citation", + "start_index": 208, + "end_index": 275, + "title": "Apache APISIX - Open Source API Gateway & AI Gateway", + "url": "https://apisix.apache.org/?utm_source=openai" + } + ] + } + ] + } + ] +} +``` diff --git a/docs/zh/latest/plugins/ai-proxy-multi.md b/docs/zh/latest/plugins/ai-proxy-multi.md index 041e7ef91243..3bf5070f3a66 100644 --- a/docs/zh/latest/plugins/ai-proxy-multi.md +++ b/docs/zh/latest/plugins/ai-proxy-multi.md @@ -56,7 +56,7 @@ import TabItem from '@theme/TabItem'; | OpenAI Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | | OpenAI Embeddings | 请求体包含 `input`,并且之前的规则均未匹配。 | 路由匹配的任意 URI。 | -基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果没有规则匹配,插件会将请求视为透传请求。 +基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果之前的规则均未匹配,插件会将非空 JSON 对象视为透传请求。空请求体或无效请求体会被拒绝。 ### Chat Completions 请求格式 @@ -92,7 +92,7 @@ import TabItem from '@theme/TabItem'; | balancer.key | string | 否 | | | 当 `type` 为 `chash` 时使用。当 `hash_on` 设置为 `header` 或 `cookie` 时,需要 `key`。当 `hash_on` 设置为 `consumer` 时,不需要 `key`,因为消费者名称将自动用作键。 | | instances | array[object] | 是 | | | LLM 实例配置。 | | instances.name | string | 是 | | | LLM 服务实例的名称。 | -| instances.provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到 `api.openai.com` 上对应的端点。设置为 `deepseek` 时,插件将代理请求到 `api.deepseek.com`。设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.aimlapi.com`。设置为 `anthropic` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.anthropic.com`。设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `openrouter.ai`。设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `generativelanguage.googleapis.com`。设置为 `vertex-ai` 时,插件默认将请求代理到 `aiplatform.googleapis.com`,且需要配置 `provider_conf` 或 `override`。设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。 | +| instances.provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到 `api.openai.com` 上对应的端点。设置为 `deepseek` 时,插件将代理请求到 `api.deepseek.com`。设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `api.aimlapi.com`。设置为 `anthropic` 时,插件会将检测到的 Chat Completions 请求发送到 `api.anthropic.com` 上的 `/v1/chat/completions`,并将原生 Anthropic Messages 请求发送到 `/v1/messages`。设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `openrouter.ai`。设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `generativelanguage.googleapis.com`。设置为 `vertex-ai` 时,插件默认将请求代理到 `aiplatform.googleapis.com`,且需要配置 `provider_conf` 或 `override`。设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。 | | instances.provider_conf | object | 否 | | | 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必填。当 `provider` 设置为 `bedrock` 时必填。 | | instances.provider_conf.project_id | string | 是 | | | Google Cloud 项目 ID。 | | instances.provider_conf.region | string | 视提供商而定 | | minLength = 1(Bedrock 时) | 当 `provider` 为 `vertex-ai` 时,此项为 Google Cloud 区域。当 `provider` 为 `bedrock` 时,此项为用于构造 Bedrock 端点并使用 SigV4 对请求进行签名的 AWS 区域(必填,不能为空)。 | diff --git a/docs/zh/latest/plugins/ai-proxy.md b/docs/zh/latest/plugins/ai-proxy.md index ae8e68bc1337..8448914cd4b8 100644 --- a/docs/zh/latest/plugins/ai-proxy.md +++ b/docs/zh/latest/plugins/ai-proxy.md @@ -56,7 +56,7 @@ import TabItem from '@theme/TabItem'; | OpenAI Chat Completions | 请求体包含 `messages` 数组。 | 路由匹配的任意 URI。 | | OpenAI Embeddings | 请求体包含 `input`,并且之前的规则均未匹配。 | 路由匹配的任意 URI。 | -基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果没有规则匹配,插件会将请求视为透传请求。 +基于 URI 的规则在仅基于请求体的规则之前运行,以免包含 `messages` 的 Bedrock Converse 和 Anthropic Messages 请求被识别为 Chat Completions。Responses 和 Embeddings 请求都使用 `input`,因此包含 `input` 但不包含 `messages` 的请求会被识别为 Embeddings,除非其 URI 以 `/v1/responses` 结尾。如果之前的规则均未匹配,插件会将非空 JSON 对象视为透传请求。空请求体或无效请求体会被拒绝。 ### Chat Completions 请求格式 @@ -83,7 +83,7 @@ import TabItem from '@theme/TabItem'; | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | |--------------------|--------|----------|---------|------------------------------------------|-------------| -| provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。当设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到对应的 OpenAI 端点。当设置为 `deepseek` 时,插件将代理请求到 `https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic` 时,插件将代理请求到 `https://api.anthropic.com/v1/chat/completions`。当设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://openrouter.ai/api/v1/chat/completions`。当设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`。当设置为 `vertex-ai` 时,插件默认将请求代理到 `https://aiplatform.googleapis.com`,需要配置 `provider_conf` 或 `override`。当设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`https://bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。当设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。当设置为 `azure-openai` 时,插件同样将请求代理到 `override` 中配置的自定义端点,并会额外移除用户请求中的 `model` 参数。 | +| provider | string | 是 | | [openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, bedrock, openai-compatible] | LLM 服务提供商。当设置为 `openai` 时,插件会将检测到的 Chat Completions、Responses 和 Embeddings 请求发送到对应的 OpenAI 端点。当设置为 `deepseek` 时,插件将代理请求到 `https://api.deepseek.com/chat/completions`。当设置为 `aimlapi` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://api.aimlapi.com/v1/chat/completions`。当设置为 `anthropic` 时,插件会将检测到的 Chat Completions 请求发送到 `https://api.anthropic.com/v1/chat/completions`,并将原生 Anthropic Messages 请求发送到 `https://api.anthropic.com/v1/messages`。当设置为 `openrouter` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://openrouter.ai/api/v1/chat/completions`。当设置为 `gemini` 时,插件使用 OpenAI 兼容驱动程序,默认将请求代理到 `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions`。当设置为 `vertex-ai` 时,插件默认将请求代理到 `https://aiplatform.googleapis.com`,需要配置 `provider_conf` 或 `override`。当设置为 `bedrock` 时,插件将代理请求到 AWS Bedrock Converse API(`https://bedrock-runtime..amazonaws.com`),并使用 AWS SigV4 对请求进行签名。当设置为 `openai-compatible` 时,插件将代理请求到在 `override` 中配置的自定义端点。当设置为 `azure-openai` 时,插件同样将请求代理到 `override` 中配置的自定义端点,并会额外移除用户请求中的 `model` 参数。 | | provider_conf | object | 否 | | | 特定提供商的配置。当 `provider` 设置为 `vertex-ai` 且未配置 `override` 时必填。当 `provider` 设置为 `bedrock` 时必填。 | | provider_conf.project_id | string | 是 | | | Google Cloud 项目 ID。 | | provider_conf.region | string | 视提供商而定 | | minLength = 1(Bedrock 时) | 当 `provider` 为 `vertex-ai` 时,此项为 Google Cloud 区域。当 `provider` 为 `bedrock` 时,此项为用于构造 Bedrock 端点并使用 SigV4 对请求进行签名的 AWS 区域(必填,不能为空)。 | diff --git a/docs/zh/latest/plugins/ai-rag.md b/docs/zh/latest/plugins/ai-rag.md index 8593f3077ada..e1833a8eea20 100644 --- a/docs/zh/latest/plugins/ai-rag.md +++ b/docs/zh/latest/plugins/ai-rag.md @@ -44,6 +44,14 @@ import TabItem from '@theme/TabItem'; 该插件使用每种协议的原生提示结构增强 Chat Completions、Responses API、Anthropic Messages 和 Bedrock Converse 请求。对于 Responses,它会将检索到的上下文追加到 `input`。它不会增强直接的 Embeddings API 请求。嵌套的 `ai_rag.embeddings` 对象用于配置检索过程中内部使用的嵌入输入,与顶层 Embeddings 请求不同。 +APISIX 会先检查基于 URI 的格式,再检查仅基于请求体的格式: + +- Bedrock Converse 要求 URI 以 `/converse` 结尾且请求体包含 `messages` 数组。 +- Anthropic Messages 要求 URI 以 `/v1/messages` 结尾。 +- Responses API 要求 URI 以 `/v1/responses` 结尾且请求体包含 `input`。 +- Chat Completions 使用 `messages` 数组。 +- 之前的规则均未匹配时,包含 `input` 的请求会被识别为 Embeddings。 + ## 插件属性 | 名称 | 类型 | 必选项 | 默认值 | 有效值 | 描述 | diff --git a/docs/zh/latest/plugins/ai-request-rewrite.md b/docs/zh/latest/plugins/ai-request-rewrite.md index 415b160b155c..e2817e9aefb9 100644 --- a/docs/zh/latest/plugins/ai-request-rewrite.md +++ b/docs/zh/latest/plugins/ai-request-rewrite.md @@ -38,7 +38,7 @@ import TabItem from '@theme/TabItem'; `ai-request-rewrite` 插件在将客户端请求转发到上游服务之前,先将请求发送到 LLM 服务进行转换处理。这使得 LLM 能够对请求进行数据脱敏、内容增强或格式转换等修改。该插件支持集成 OpenAI、DeepSeek、Gemini、Vertex AI、Anthropic、OpenRouter 以及其他 OpenAI 兼容的 API。 -用于重写的 LLM 调用与客户端请求格式相互独立。使用 `openai` 提供商时,插件会根据配置的提示词和原始请求体构建非流式 Chat Completions 请求。它不会使用 [`ai-proxy` 请求协议检测](./ai-proxy.md#请求协议检测)。 +用于重写的 LLM 调用与客户端请求格式相互独立。使用 `openai` 提供商时,插件会根据配置的提示词和原始请求体构建非流式 Chat Completions 请求。它不会将客户端请求识别为 AI 协议,也不会按 AI 协议代理该请求。 ## 插件属性