diff --git a/app/_data/ai-gateway/v2/otel-metrics.yaml b/app/_data/ai-gateway/v2/otel-metrics.yaml new file mode 100644 index 00000000000..9b8da8980f0 --- /dev/null +++ b/app/_data/ai-gateway/v2/otel-metrics.yaml @@ -0,0 +1,332 @@ +metrics: + - name: gen_ai.client.operation.duration + min_version: "" + description: Total time Kong spends processing a Gen AI operation, such as an LLM request. Requires `enable_request_metrics` to populate the `error.type` attribute. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - error.type + - name: gen_ai.server.request.duration + min_version: "" + description: Time the LLM provider spends processing the request. Requires `enable_latency_metrics` set to `true`. Requires `enable_request_metrics` to populate the `error.type` attribute. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - error.type + - name: gen_ai.client.token.usage + min_version: "" + description: Number of tokens consumed by the Gen AI operation. + unit: "{token}" + type: Sum + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.token.type + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: gen_ai.server.time_to_first_token + min_version: "" + description: Time from when the model server receives the request until the first output token is generated. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: gen_ai.server.time_per_output_token + min_version: "" + description: Time between successive output tokens generated by the model server after the first token. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.llm.cost + min_version: "" + description: Cost of AI requests. + unit: "{cost}" + type: Sum + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.cache.fetch.latency + min_version: "" + description: Time to fetch a response from the semantic cache. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.cache.embeddings.latency + min_version: "" + description: Time to generate embeddings during cache operations. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.rag.fetch.latency + min_version: "" + description: Time to fetch data from a RAG source. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.rag.embeddings.latency + min_version: "" + description: Time to generate embeddings for RAG operations. + unit: "s" + type: Histogram + attributes: + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.response.model + - gen_ai.operation.name + - kong.gen_ai.cache.status + - kong.gen_ai.vector_db + - kong.gen_ai.embeddings.provider + - kong.gen_ai.embeddings.model + - kong.workspace.name + - kong.auth.consumer.name + - kong.gen_ai.request.mode + - name: kong.gen_ai.aws.guardrails.latency + min_version: "" + description: Time for AWS Guardrails to process a request. + unit: "s" + type: Histogram + attributes: + - kong.gen_ai.aws.guardrails.id + - kong.gen_ai.aws.guardrails.version + - kong.gen_ai.aws.guardrails.mode + - kong.gen_ai.aws.guardrails.region + - kong.workspace.name + - kong.auth.consumer.name + - name: kong.gen_ai.mcp.response.size + min_version: "" + description: Size of the MCP response body in bytes. + unit: "By" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - name: kong.gen_ai.mcp.request.error.count + min_version: "" + description: Number of MCP request errors. + unit: "{error}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - name: mcp.client.operation.duration + min_version: "" + description: Duration of the MCP request as observed by the sender. Only available when the MCP entity is in passthrough-listener mode. Requires `enable_latency_metrics` set to `true`. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - gen_ai.operation.name + - name: mcp.server.operation.duration + min_version: "" + description: Duration of the MCP request as observed by the receiver. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - mcp.method.name + - gen_ai.tool.name + - error.type + - gen_ai.operation.name + - name: kong.gen_ai.mcp.acl.allowed + min_version: "" + description: Number of MCP requests allowed by ACL rules. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.mcp.primitive + - kong.gen_ai.mcp.primitive_name + - name: kong.gen_ai.mcp.acl.denied + min_version: "" + description: Number of MCP requests denied by ACL rules. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.mcp.primitive + - kong.gen_ai.mcp.primitive_name + - name: kong.gen_ai.a2a.request.count + min_version: "" + description: Total number of A2A requests. + unit: "{request}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.request.duration + min_version: "" + description: Duration of an A2A request in seconds. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.response.size + min_version: "" + description: Size of the A2A response body in bytes. + unit: "By" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.ttfb + min_version: "" + description: Time to first byte for A2A streaming responses in seconds. + unit: "s" + type: Histogram + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - name: kong.gen_ai.a2a.request.error.count + min_version: "" + description: Number of A2A request errors. + unit: "{error}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.method + - kong.gen_ai.a2a.binding + - kong.gen_ai.a2a.error.type + - name: kong.gen_ai.a2a.task.state.count + min_version: "" + description: Number of A2A task state transitions. + unit: "{state}" + type: Sum + attributes: + - kong.service.name + - kong.route.name + - kong.workspace.name + - kong.gen_ai.a2a.task.state + +attributes: + kong.service.name: Name of the Gateway Service. + kong.route.name: Name of the Route. + kong.auth.consumer.name: Name of the authenticated Consumer. + kong.workspace.name: Name of the Workspace. + error.type: Type of error that occurred. + gen_ai.provider.name: Name of the Gen AI provider. + gen_ai.request.model: Model name targeted by the request. + gen_ai.response.model: Model name reported by the provider in the response. + gen_ai.operation.name: "Operation requested, such as `chat` or `embeddings`." + gen_ai.token.type: "Token category: `input`, `output`, or `total`." + kong.gen_ai.request.mode: "Request mode: `oneshot`, `stream`, or `realtime`." + kong.gen_ai.cache.status: "Cache status: `hit` or empty if not cached." + kong.gen_ai.vector_db: "Vector database used for caching, such as `redis`." + kong.gen_ai.embeddings.provider: Embeddings provider used for caching. + kong.gen_ai.embeddings.model: Embeddings model used for caching. + kong.gen_ai.aws.guardrails.id: ID of the AWS Guardrails configuration. + kong.gen_ai.aws.guardrails.version: Version of the AWS Guardrails configuration. + kong.gen_ai.aws.guardrails.mode: Mode of the AWS Guardrails evaluation. + kong.gen_ai.aws.guardrails.region: AWS region of the Guardrails service. + mcp.method.name: "MCP method name, such as `tools/call`." + gen_ai.tool.name: Name of the MCP tool invoked. + kong.gen_ai.mcp.primitive: "MCP primitive type, such as `tool`." + kong.gen_ai.mcp.primitive_name: Name of the MCP primitive. + kong.gen_ai.a2a.method: A2A method name. + kong.gen_ai.a2a.binding: A2A binding type. + kong.gen_ai.a2a.error.type: Type of the A2A error. + kong.gen_ai.a2a.task.state: "Task state, such as `completed`, `failed`, or `in_progress`." diff --git a/app/_data/ai-gateway/v2/otel-span-attributes.yaml b/app/_data/ai-gateway/v2/otel-span-attributes.yaml new file mode 100644 index 00000000000..a4c73882637 --- /dev/null +++ b/app/_data/ai-gateway/v2/otel-span-attributes.yaml @@ -0,0 +1,75 @@ +spans: + - name: kong.gen_ai + title: Gen AI span attributes + min_version: "" + description: Gen AI tracing span emitted for LLM requests. + attributes: + - gen_ai.operation.name + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.request.max_tokens + - gen_ai.request.temperature + - gen_ai.input.messages + - gen_ai.output.type + - gen_ai.output.messages + - gen_ai.response.id + - gen_ai.response.model + - gen_ai.response.finish_reasons + - gen_ai.usage.input_tokens + - gen_ai.usage.output_tokens + - name: kong.gen_ai + title: Gen AI tool call span attributes + min_version: "" + description: Gen AI tracing span emitted when the provider response includes a tool call. + attributes: + - gen_ai.operation.name + - gen_ai.provider.name + - gen_ai.request.model + - gen_ai.request.max_tokens + - gen_ai.request.temperature + - gen_ai.response.finish_reasons + - gen_ai.response.id + - gen_ai.response.model + - gen_ai.tool.call.id + - gen_ai.tool.name + - gen_ai.tool.type + - gen_ai.usage.input_tokens + - gen_ai.usage.output_tokens + - gen_ai.output.type + - name: kong.a2a + title: A2A span attributes + min_version: "" + description: A2A tracing span emitted for agent-to-agent requests. + attributes: + - kong.a2a.protocol.version + - rpc.system + - rpc.method + - kong.a2a.task.id + - kong.a2a.task.state + - kong.a2a.context.id + - kong.a2a.operation + +attributes: + gen_ai.operation.name: "Operation requested, such as `chat` or `embeddings`." + gen_ai.provider.name: Name of the Gen AI provider. + gen_ai.request.model: Model name targeted by the request. + gen_ai.request.max_tokens: Maximum token limit configured for the request. + gen_ai.request.temperature: Sampling temperature configured for the request. + gen_ai.input.messages: Array of input messages sent to the model. + gen_ai.output.type: Output payload type, such as `json`. + gen_ai.output.messages: Array containing the full model response payload. + gen_ai.response.id: Unique identifier returned by the provider for the response. + gen_ai.response.model: Model name reported by the provider in the response. + gen_ai.response.finish_reasons: Array of finish reasons returned by the provider. + gen_ai.usage.input_tokens: Number of input tokens consumed by the request. + gen_ai.usage.output_tokens: Number of output tokens generated in the response. + gen_ai.tool.call.id: Unique identifier for the specific tool call. + gen_ai.tool.name: Name of the tool or function requested by the model. + gen_ai.tool.type: Tool type, such as `function`. + kong.a2a.protocol.version: A2A protocol version used for the request. + rpc.system: RPC protocol used by the request, such as `jsonrpc`. + rpc.method: RPC method invoked by the client. + kong.a2a.task.id: Identifier of the A2A task. + kong.a2a.task.state: Current state of the A2A task. + kong.a2a.context.id: Identifier of the A2A conversation context. + kong.a2a.operation: A2A operation name, such as `message/send`. diff --git a/app/_includes/md/ai-gateway/v2/policies/collecting-otel-data.md b/app/_includes/md/ai-gateway/v2/policies/collecting-otel-data.md new file mode 100644 index 00000000000..39681c13ddc --- /dev/null +++ b/app/_includes/md/ai-gateway/v2/policies/collecting-otel-data.md @@ -0,0 +1,19 @@ +## Collecting telemetry data + +To set up an OpenTelemetry backend for {{site.ai_gateway}}, you need support for OTLP over HTTP with Protobuf encoding. You can: + +* Send data directly to an OpenTelemetry-compatible backend that natively supports OTLP over HTTP with Protobuf encoding, like Jaeger (v1.35.0+). + + This is the simplest setup, since it doesn't require any additional components between the data plane and the backend. + +* Use the OpenTelemetry Collector, which acts as an intermediary between the data plane and one or more backends. + + OTEL Collector can receive all OpenTelemetry signals supported by the {{site.ai_gateway}} OpenTelemetry Policy, including traces, metrics, and logs, and then process, transform, or route that data before exporting it to a compatible backend. + + This option is useful when you need capabilities such as signal fan-out, filtering, enrichment, batching, or exporting to multiple backends. The OpenTelemetry Collector supports a wide range of exporters, available at [open-telemetry/opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter). + +{% assign policy = include.policy | default: "default" %} +{% unless policy == "OpenTelemetry" %} +{:.info} +> Check [OpenTelemetry Policy](/ai-gateway/entities/ai-opentelemetry-policy/) and [{{site.base_gateway}} tracing](/gateway/tracing/) documentation for more details about OpenTelemetry and tracing in {{site.ai_gateway}}. +{% endunless %} diff --git a/app/_includes/md/ai-gateway/v2/policies/metric_tables.md b/app/_includes/md/ai-gateway/v2/policies/metric_tables.md new file mode 100644 index 00000000000..a0c6ec13f44 --- /dev/null +++ b/app/_includes/md/ai-gateway/v2/policies/metric_tables.md @@ -0,0 +1,43 @@ +{%- assign metrics = site.data.ai-gateway.v2.otel-metrics.metrics -%} +{%- assign attributes = site.data.ai-gateway.v2.otel-metrics.attributes -%} +{% for metric in metrics %} +{% if include.metric_prefixes %} +{% assign found = false %} +{% assign prefixes = include.metric_prefixes | split: ',' %} +{% for prefix in prefixes %} +{% assign prefix_stripped = prefix | strip %} +{% assign metric_prefix = metric.name | slice: 0, prefix_stripped.size | strip %} +{% if metric_prefix == prefix_stripped %} +{% assign found = true %} +{% break %} +{% endif %} +{% endfor %} +{% unless found %}{% continue %}{% endunless %} +{% endif %} +#### {{metric.name}}{% if metric.min_version != "" %} {% new_in metric.min_version %}{% endif %} + +{{metric.description}} + +{% if metric.unit %}- **Instrument unit**: `{{metric.unit}}`{% endif %} +{% if metric.type %}- **Instrument type**: `{{metric.type}}`{% endif %} +{% if metric.attributes %}- **Attributes**: +{% capture attrs_table %} +{% table %} +vertical_align: middle +columns: + - title: Attribute + key: attribute + - title: Attribute description + key: description +rows: +{% for attribute in metric.attributes %} + - id: "{{attribute}}" + attribute: "`{{ attribute }}`" + description: | +{{attributes[attribute] | indent: 6}} +{% endfor %} +{% endtable %} +{% endcapture %} +{{attrs_table | indent: 2}} +{% else %}- **No attributes**{% endif %} +{% endfor %} diff --git a/app/_includes/md/ai-gateway/v2/policies/span_attribute_tables.md b/app/_includes/md/ai-gateway/v2/policies/span_attribute_tables.md new file mode 100644 index 00000000000..838ec60066e --- /dev/null +++ b/app/_includes/md/ai-gateway/v2/policies/span_attribute_tables.md @@ -0,0 +1,30 @@ +{%- assign spans = site.data.ai-gateway.v2.otel-span-attributes.spans -%} +{%- assign attributes = site.data.ai-gateway.v2.otel-span-attributes.attributes -%} +{% for span in spans %} +#### {{ span.title | default: span.name }}{% if span.min_version != "" %} {% new_in span.min_version %}{% endif %} + +{{ span.description }} + +{% if span.title and span.title != span.name %} The following span attributes use the `{{ span.name }}` prefix{% if span.name == "kong.a2a" %} or the `rpc` prefix{% endif %}:{% endif %} + + +{% capture attrs_table %} +{% table %} +vertical_align: middle +columns: + - title: Attribute + key: attribute + - title: Attribute description + key: description +rows: +{% for attribute in span.attributes %} + - id: "{{attribute}}" + attribute: "`{{ attribute }}`" + description: | +{{attributes[attribute] | indent: 6}} +{% endfor %} +{% endtable %} +{% endcapture %} +{{attrs_table | indent: 2}} +{% endfor %} + diff --git a/app/_includes/plugins/ai-a2a-proxy/log-output-fields.md b/app/_includes/plugins/ai-a2a-proxy/log-output-fields.md index 240d0295692..6d9a9533b73 100644 --- a/app/_includes/plugins/ai-a2a-proxy/log-output-fields.md +++ b/app/_includes/plugins/ai-a2a-proxy/log-output-fields.md @@ -1,4 +1,4 @@ -When `config.logging.log_statistics` is enabled, the plugin writes the following fields to the +When `config.logging.log_statistics` is enabled, it writes the following fields to the `ai.a2a.rpc[]` array: {% table %} diff --git a/app/ai-gateway/ai-audit-log-reference.md b/app/ai-gateway/ai-audit-log-reference.md index bf9a084c71f..5986646bc85 100644 --- a/app/ai-gateway/ai-audit-log-reference.md +++ b/app/ai-gateway/ai-audit-log-reference.md @@ -5,45 +5,44 @@ layout: reference products: - ai-gateway - - gateway tags: - ai - logging min_version: - gateway: '3.6' + ai-gateway: '2.0' breadcrumbs: - /ai-gateway/ -description: "{{site.ai_gateway}} provides a standardized logging format for AI plugins, enabling the emission of analytics events and facilitating the aggregation of AI usage analytics across various providers." +description: "{{site.ai_gateway}} provides a standardized logging format for AI Policies, enabling the emission of analytics events and facilitating the aggregation of AI usage analytics across various providers." related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: "{{site.ai_gateway}} plugins" - url: /plugins/?category=ai - - text: "{{site.base_gateway}} logs" - url: /gateway/logs/ + - text: "{{site.ai_gateway}} logs" + url: /ai-gateway/ai-logs/ works_on: - - on-prem - konnect --- -{{site.ai_gateway}} emits structured analytics logs for [AI plugins](/plugins/?category=ai) through the standard [{{site.base_gateway}} logging infrastructure](/gateway/logs/). This means AI-specific logs are written to [the same locations](/gateway/logs/#where-are-kong-gateway-logs-located) as other Kong logs, such as `/usr/local/kong/logs/error.log`, or to Docker container logs if you're running in a containerized environment. +{{site.ai_gateway}} emits structured analytics logs for [AI Policies](/plugins/?category=ai) following the same patterns as {{site.base_gateway}}. This means {{site.ai_gateway}} logs are written to [the same locations](/ai-gateway/ai-logs/#where-are-ai-gateway-logs-located) as other Kong logs, such as `/usr/local/kong/logs/error.log`, or to Docker container logs if you're running in a containerized environment. -Like other Kong logs, {{site.ai_gateway}} logs are subject to the [global log level](/gateway/logs/#configure-log-levels) configured via the [`kong.conf`](/gateway/configuration/) file or the Admin API. You can control log verbosity by adjusting the `log_level` setting (for example, `info`, `notice`, `warn`, `error`, `crit`) to determine which log entries are captured. +You can set the [global log level](/ai-gateway/ai-logs/#configure-log-levels) for {{site.ai_gateway}} via the [`kong.conf`](/gateway/configuration/) file or the Admin API. You can control log verbosity by adjusting the `log_level` setting (for example, `info`, `notice`, `warn`, `error`, `crit`) to determine which log entries are captured. -You can also use [logging plugins](/plugins/?category=logging) to route these logs to external systems, such as file systems, log aggregators, or monitoring tools. +When operating {{site.ai_gateway}} alongside {{site.base_gateway}}, logs are stored separately in each product's run time environment. + +You can also use [logging Policies](/plugins/?category=logging) to route these logs to external systems, such as file systems, log aggregators, or monitoring tools. ## Log details -Each AI plugin returns a set of tokens. Log entries include the following details: +Each {{site.ai_gateway}} policy returns a set of tokens. Log entries include the following details: +### Core logs -### AI Proxy core logs +{{site.ai_gateway}} logs capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics. They provide a comprehensive view of each AI interaction. -The [AI Proxy](/plugins/ai-proxy/) and [AI Proxy Advanced](/plugins/ai-proxy-advanced/) plugins act as the main gateway for forwarding requests to AI providers. Logs here capture detailed information about the request and response payloads, token usage, model details, latency, and cost metrics. They provide a comprehensive view of each AI interaction. +The core proxy functionality is provided by the [AI Proxy](/plugins/ai-proxy/) and [AI Proxy Advanced](/plugins/ai-proxy-advanced/) which is reflected in the property names. {:.warning} > Logs and metrics for cost and token usage via the [OpenAI Files API](https://developers.openai.com/api/reference/resources/files/methods/list) are not currently supported. @@ -65,40 +64,40 @@ rows: Used for text-based requests (chat, completions, embeddings). - property: "`ai.proxy.usage.prompt_tokens_details`" description: | - {% new_in 3.11 %} A breakdown of prompt tokens (`cached_tokens`, `audio_tokens`). + A breakdown of prompt tokens (`cached_tokens`, `audio_tokens`). - property: "`ai.proxy.usage.completion_tokens`" description: | The number of tokens used for completion. Used for text-based responses (chat, completions). - property: "`ai.proxy.usage.completion_tokens_details`" description: | - {% new_in 3.11 %} A breakdown of completion tokens (`rejected_prediction_tokens`, `reasoning_tokens`, `accepted_prediction_tokens`, `audio_tokens`). + A breakdown of completion tokens (`rejected_prediction_tokens`, `reasoning_tokens`, `accepted_prediction_tokens`, `audio_tokens`). - property: "`ai.proxy.usage.total_tokens`" description: | The total number of tokens used (input + output). Includes prompt/completion tokens for text, and input/output tokens for non-text modalities. - property: "`ai.proxy.usage.input_tokens`" description: | - {% new_in 3.11 %} The total number of input tokens (text + image + audio). + The total number of input tokens (text + image + audio). Used for non-text requests (e.g., image or audio generation). - property: "`ai.proxy.usage.input_tokens_details`" description: | - {% new_in 3.11 %} A breakdown of input tokens by modality (`text_tokens`, `image_tokens`, `audio_tokens_count`). + A breakdown of input tokens by modality (`text_tokens`, `image_tokens`, `audio_tokens_count`). - property: "`ai.proxy.usage.output_tokens`" description: | - {% new_in 3.11 %} The total number of output tokens (text + audio). + The total number of output tokens (text + audio). Used for non-text responses (e.g., image or audio generation). - property: "`ai.proxy.usage.output_tokens_details`" description: | - {% new_in 3.11 %} A breakdown of output tokens by modality (`text_tokens`, `audio_tokens`). + A breakdown of output tokens by modality (`text_tokens`, `audio_tokens`). - property: "`ai.proxy.usage.cost`" description: The total cost of the request. - property: "`ai.proxy.usage.time_per_token`" description: | - {% new_in 3.8 %} Average time to generate an output token (ms). + Average time to generate an output token (ms). - property: "`ai.proxy.usage.time_to_first_token`" description: | - {% new_in 3.12 %} Time to receive the first output token (ms). + Time to receive the first output token (ms). - property: "`ai.proxy.meta.request_model`" description: The model used for the AI request. - property: "`ai.proxy.meta.response_model`" @@ -106,18 +105,18 @@ rows: - property: "`ai.proxy.meta.provider_name`" description: The name of the AI service provider. - property: "`ai.proxy.meta.plugin_id`" - description: Unique identifier of the plugin instance. + description: Unique identifier of the Policy instance. - property: "`ai.proxy.meta.llm_latency`" description: | - {% new_in 3.8 %} Time taken by the LLM provider to generate the full response (ms). + Time taken by the LLM provider to generate the full response (ms). - property: "`ai.proxy.meta.request_mode`" description: | - {% new_in 3.12 %} The request mode. Can be `oneshot`, `stream`, or `realtime`. + The request mode. Can be `oneshot`, `stream`, or `realtime`. {% endtable %} -### AI AWS Guardrails logs {% new_in 3.11 %} +### AI AWS Guardrails logs -If you're using the [AI AWS Guardrails plugin](/plugins/ai-aws-guardrails/), {{site.ai_gateway}} logs include fields under the `ai.proxy.aws-guardrails` object. These fields capture processing latency, the guardrails configuration applied, block reasons, and masking behavior. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI AWS Guardrails Policy](/plugins/ai-aws-guardrails/), {{site.ai_gateway}} logs include fields under the `ai.proxy.aws-guardrails` object. These fields capture processing latency, the guardrails configuration applied, block reasons, and masking behavior. {% table %} columns: @@ -134,7 +133,7 @@ rows: description: "The version of the guardrail applied. Can be a numeric version or `DRAFT`." - property: "`ai.proxy.aws-guardrails.mode`" description: | - {% new_in 3.14 %} The content guarding mode configured for the plugin. Possible values: `INPUT`, `OUTPUT`, `BOTH`. + The content guarding mode configured for the Policy. Possible values: `INPUT`, `OUTPUT`, `BOTH`. - property: "`ai.proxy.aws-guardrails.input_processing_latency`" description: The time, in milliseconds, spent processing the request through the guardrail. - property: "`ai.proxy.aws-guardrails.output_processing_latency`" @@ -149,30 +148,30 @@ rows: description: "`true` if the response content was masked rather than blocked. Only present when `config.allow_masking` is `true`." - property: "`ai.proxy.aws-guardrails.input_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the request. Empty if the request was allowed. + The name of the Policy that blocked the request. Empty if the request was allowed. - property: "`ai.proxy.aws-guardrails.output_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the response. Empty if the response was allowed. + The name of the Policy that blocked the response. Empty if the response was allowed. - property: "`ai.proxy.aws-guardrails.input_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.aws-guardrails.output_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.aws-guardrails.guards_triggered_count`" description: | - {% new_in 3.14 %} A counter that increments each time a block is triggered on either the input or output within a single request. + A counter that increments each time a block is triggered on either the input or output within a single request. - property: "`ai.proxy.aws-guardrails.input_faulty_prompt`" description: | - {% new_in 3.14 %} The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. - property: "`ai.proxy.aws-guardrails.output_faulty_response`" description: | - {% new_in 3.14 %} The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. {% endtable %} -### AI GCP Model Armor logs {% new_in 3.12 %} +### AI GCP Model Armor logs -If you're using the [AI GCP Model Armor plugin](/plugins/ai-gcp-model-armor/), {{site.ai_gateway}} logs include fields under the `ai.proxy.gcp-model-armor` object. These fields capture the template applied, processing latency, and reasons for blocking when content is flagged. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI GCP Model Armor Policy](/plugins/ai-gcp-model-armor/), {{site.ai_gateway}} logs include fields under the `ai.proxy.gcp-model-armor` object. These fields capture the template applied, processing latency, and reasons for blocking when content is flagged. {% table %} columns: @@ -193,33 +192,33 @@ rows: description: "The check type or types that caused the response to be blocked, comma-separated. Empty if the response was allowed." - property: "`ai.proxy.gcp-model-armor.mode`" description: | - {% new_in 3.14 %} The content guarding mode configured for the plugin. Possible values: `INPUT`, `OUTPUT`, `BOTH`. + The content guarding mode configured for the Policy. Possible values: `INPUT`, `OUTPUT`, `BOTH`. - property: "`ai.proxy.gcp-model-armor.input_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the request. Empty if the request was allowed. + The name of the Policy that blocked the request. Empty if the request was allowed. - property: "`ai.proxy.gcp-model-armor.output_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the response. Empty if the response was allowed. + The name of the Policy that blocked the response. Empty if the response was allowed. - property: "`ai.proxy.gcp-model-armor.input_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.gcp-model-armor.output_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.gcp-model-armor.guards_triggered_count`" description: | - {% new_in 3.14 %} A counter that increments each time a block is triggered on either the input or output within a single request. + A counter that increments each time a block is triggered on either the input or output within a single request. - property: "`ai.proxy.gcp-model-armor.input_faulty_prompt`" description: | - {% new_in 3.14 %} The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. - property: "`ai.proxy.gcp-model-armor.output_faulty_response`" description: | - {% new_in 3.14 %} The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. {% endtable %} ### AI Azure Content Safety logs -If you're using the [AI Azure Content Safety plugin](/plugins/ai-azure-content-safety/), {{site.ai_gateway}} writes to two separate log paths. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI Azure Content Safety Policy](/plugins/ai-azure-content-safety/), {{site.ai_gateway}} writes to two separate log paths. The first path records per-category severity data from the Azure Content Safety API. Each entry represents a category that breached its configured rejection threshold. Multiple entries can appear per request depending on which categories were configured and what was detected. @@ -236,7 +235,7 @@ rows: description: "The numeric rejection severity threshold for the category that was breached (for example, `Hate`, `Violence`). Defined by `config.categories[*].rejection_level`. Multiple entries can appear per request." {% endtable %} -The second path records plugin metadata and block reasons under the `ai.proxy.azure-content-safety` object: +The second path records Policy metadata and block reasons under the `ai.proxy.azure-content-safety` object: {% table %} columns: @@ -263,33 +262,33 @@ rows: description: The reason the response was blocked. Empty if the response was allowed. - property: "`ai.proxy.azure-content-safety.mode`" description: | - {% new_in 3.14 %} The content guarding mode configured for the plugin. Possible values: `INPUT`, `OUTPUT`, `BOTH`. + The content guarding mode configured for the Policy. Possible values: `INPUT`, `OUTPUT`, `BOTH`. - property: "`ai.proxy.azure-content-safety.input_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the request. Empty if the request was allowed. + The name of the Policy that blocked the request. Empty if the request was allowed. - property: "`ai.proxy.azure-content-safety.output_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the response. Empty if the response was allowed. + The name of the Policy that blocked the response. Empty if the response was allowed. - property: "`ai.proxy.azure-content-safety.input_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.azure-content-safety.output_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.azure-content-safety.guards_triggered_count`" description: | - {% new_in 3.14 %} A counter that increments each time a block is triggered on either the input or output within a single request. + A counter that increments each time a block is triggered on either the input or output within a single request. - property: "`ai.proxy.azure-content-safety.input_faulty_prompt`" description: | - {% new_in 3.14 %} The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. - property: "`ai.proxy.azure-content-safety.output_faulty_response`" description: | - {% new_in 3.14 %} The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. {% endtable %} -### AI Lakera Guard logs {% new_in 3.13 %} +### AI Lakera Guard logs -If you're using the [AI Lakera Guard plugin](/plugins/ai-lakera-guard/), {{site.ai_gateway}} logs include additional fields under the `ai.proxy.lakera-guard` object. These fields capture processing latency, Lakera-assigned request UUIDs, block reasons, and violation details when requests or responses are blocked. +If you create an [ AI Policy](/ai-gateway/entities/ai-policy/) using the [AI Lakera Guard Policy](/plugins/ai-lakera-guard/), {{site.ai_gateway}} logs include additional fields under the `ai.proxy.lakera-guard` object. These fields capture processing latency, Lakera-assigned request UUIDs, block reasons, and violation details when requests or responses are blocked. {% table %} columns: @@ -304,7 +303,7 @@ rows: description: "The Lakera project identifier used for the inspection. Defaults to `default` if no project ID is configured." - property: "`ai.proxy.lakera-guard.mode`" description: | - {% new_in 3.14 %} The content guarding mode configured for the plugin. Possible values: `INPUT`, `OUTPUT`, `BOTH`. + The content guarding mode configured for the Policy. Possible values: `INPUT`, `OUTPUT`, `BOTH`. - property: "`ai.proxy.lakera-guard.input_processing_latency`" description: The time, in milliseconds, that Lakera took to process the request. - property: "`ai.proxy.lakera-guard.output_processing_latency`" @@ -323,32 +322,32 @@ rows: description: "An array of violation objects present when Lakera blocks a response. The structure matches `input_block_detail`." - property: "`ai.proxy.lakera-guard.input_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the request. Empty if the request was allowed. + The name of the Policy that blocked the request. Empty if the request was allowed. - property: "`ai.proxy.lakera-guard.output_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the response. Empty if the response was allowed. + The name of the Policy that blocked the response. Empty if the response was allowed. - property: "`ai.proxy.lakera-guard.input_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.lakera-guard.output_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.proxy.lakera-guard.guards_triggered_count`" description: | - {% new_in 3.14 %} A counter that increments each time a block is triggered on either the input or output within a single request. + A counter that increments each time a block is triggered on either the input or output within a single request. - property: "`ai.proxy.lakera-guard.input_faulty_prompt`" description: | - {% new_in 3.14 %} The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw request prompt that was blocked. Only present when `config.log_blocked_content` is `true`. - property: "`ai.proxy.lakera-guard.output_faulty_response`" description: | - {% new_in 3.14 %} The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. + The raw response that was blocked. Only present when `config.log_blocked_content` is `true`. {% endtable %} -### AI Custom Guardrail logs {% new_in 3.14 %} +### AI Custom Guardrail logs -If you're using the [AI Custom Guardrail plugin](/plugins/ai-custom-guardrail/), {{site.ai_gateway}} logs include additional fields under the `custom-guardrail` object. These fields record guardrail processing latency, block reasons, and the source and consumer identity associated with any triggered guards. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI Custom Guardrail Policy](/plugins/ai-custom-guardrail/), {{site.ai_gateway}} logs include additional fields under the `custom-guardrail` object. These fields record guardrail processing latency, block reasons, and the source and consumer identity associated with any triggered guards. -The following fields appear in structured AI logs when the AI Custom Guardrail plugin is enabled: +The following fields appear in structured AI logs when the AI Custom Guardrail Policy is enabled: {% table %} columns: @@ -381,12 +380,12 @@ rows: {% endtable %} {:.info} -> The plugin also allows you to define [custom metrics](/plugins/ai-custom-guardrail/#metrics) based on Lua expressions. +> The Policy also allows you to define [custom metrics](/plugins/ai-custom-guardrail/#metrics) based on Lua expressions. -### AI PII Sanitizer logs {% new_in 3.10 %} +### AI PII Sanitizer logs -If you're using the [AI PII Sanitizer plugin](/plugins/ai-sanitizer/), {{site.ai_gateway}} logs include additional fields that provide insight into the detection and redaction of personally identifiable information (PII). These fields track the number of entities identified and sanitized, the time taken to process the payload, and detailed metadata about each sanitized item, including the original value, redacted value, and detected entity type. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI PII Sanitizer Policy](/plugins/ai-sanitizer/), {{site.ai_gateway}} logs include additional fields that provide insight into the detection and redaction of personally identifiable information (PII). These fields track the number of entities identified and sanitized, the time taken to process the payload, and detailed metadata about each sanitized item, including the original value, redacted value, and detected entity type. {% table %} columns: @@ -405,24 +404,24 @@ rows: description: A list of sanitized PII entities, each including the original text, redacted text, and the entity type. - property: "`ai.sanitizer.input_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the request. Empty if the request was allowed. + The name of the Policy that blocked the request. Empty if the request was allowed. - property: "`ai.sanitizer.output_block_source`" description: | - {% new_in 3.14 %} The name of the plugin that blocked the response. Empty if the response was allowed. + The name of the Policy that blocked the response. Empty if the response was allowed. - property: "`ai.sanitizer.input_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose request was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.sanitizer.output_block_consumer_id`" description: | - {% new_in 3.14 %} The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. + The ID of the consumer whose response was blocked, or `unknown` if no consumer identity was resolved. - property: "`ai.sanitizer.guards_triggered_count`" description: | - {% new_in 3.14 %} A counter that increments each time a block is triggered on either the input or output within a single request. + A counter that increments each time a block is triggered on either the input or output within a single request. {% endtable %} -### AI Prompt Compressor logs {% new_in 3.11 %} +### AI Prompt Compressor logs -When the [AI Prompt Compressor plugin](/plugins/ai-prompt-compressor/) is enabled, additional logs record token counts before and after compression, compression ratios, and metadata about the compression method and model used. +When the [AI Prompt Compressor Policy](/plugins/ai-prompt-compressor/) is enabled, additional logs record token counts before and after compression, compression ratios, and metadata about the compression method and model used. {% table %} columns: @@ -449,9 +448,9 @@ rows: description: A summary or message describing the result of compression. {% endtable %} -### AI RAG Injector logs {% new_in 3.10 %} +### AI RAG Injector logs -If you're using the [AI RAG Injector plugin](/plugins/ai-rag-injector/), {{site.ai_gateway}} logs include additional fields that provide detailed information about the retrieval-augmented generation process. These fields track the vector database used, whether relevant context was injected into the prompt, the latency of data fetching, and embedding metadata such as tokens used and the embedding provider and model used. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI RAG Injector Policy](/plugins/ai-rag-injector/), {{site.ai_gateway}} logs include additional fields that provide detailed information about the retrieval-augmented generation process. These fields track the vector database used, whether relevant context was injected into the prompt, the latency of data fetching, and embedding metadata such as tokens used and the embedding provider and model used. {% table %} columns: @@ -478,9 +477,8 @@ rows: description: Model used to generate embeddings. {% endtable %} -### AI Semantic Cache logs {% new_in 3.8 %} - -If you're using the [AI Semantic Cache plugin](/plugins/ai-semantic-cache/), {{site.ai_gateway}} logs include additional fields under the cache object for each plugin entry. These fields provide insight into cache behavior, such as whether a response was served from cache, how long it took to fetch, and which embedding provider and model were used if applicable. +### AI Semantic Cache logs +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI Semantic Cache Policy](/plugins/ai-semantic-cache/), {{site.ai_gateway}} logs include additional fields under the cache object for each Policy entry. These fields provide insight into cache behavior, such as whether a response was served from cache, how long it took to fetch, and which embedding provider and model were used if applicable. {% table %} columns: @@ -491,7 +489,7 @@ columns: rows: - property: "`ai.proxy.cache.cache_status`" description: | - {% new_in 3.8 %} The cache status. This can be `Hit`, `Miss`, `Bypass`, or `Refresh`. + The cache status. This can be `Hit`, `Miss`, `Bypass`, or `Refresh`. - property: "`ai.proxy.cache.fetch_latency`" description: The time, in milliseconds, it took to return a cached response. - property: "`ai.proxy.cache.embeddings_provider`" @@ -506,9 +504,9 @@ rows: > **Note:** When returning a cached response, `time_per_token` and `llm_latency` are omitted. > The cache response can be returned either as a semantic cache or an exact cache. If it's returned as a semantic cache, it will include additional details such as the embeddings provider, embeddings model, and embeddings latency. -### AI LLM as Judge logs {% new_in 3.12 %} +### AI LLM as Judge logs -If you're using the [AI LLM as Judge plugin](/plugins/ai-llm-as-judge/), {{site.ai_gateway}} logs include additional fields under the `ai-llm-as-judge` object. These fields provide insight into evaluation behavior, such as which models were scored, latency, and the numeric accuracy assigned by the judge. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI LLM as Judge Policy](/plugins/ai-llm-as-judge/), {{site.ai_gateway}} logs include additional fields under the `ai-llm-as-judge` object. These fields provide insight into evaluation behavior, such as which models were scored, latency, and the numeric accuracy assigned by the judge. {% table %} columns: @@ -532,14 +530,14 @@ rows: {% endtable %} -### AI MCP logs {% new_in 3.12 %} +### AI MCP logs -If you're using the [AI MCP plugin](/plugins/ai-mcp-proxy/), {{site.ai_gateway}} logs include additional fields under the `ai.mcp` object. These fields provide insight into Model Context Protocol (MCP) traffic, including session IDs, JSON-RPC request/response payloads, latency, tool usage, and {% new_in 3.13 %} access control audit entries. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI MCP Policy](/plugins/ai-mcp-proxy/), {{site.ai_gateway}} logs include additional fields under the `ai.mcp` object. These fields provide insight into Model Context Protocol (MCP) traffic, including session IDs, JSON-RPC request/response payloads, latency, tool usage, and access control audit entries. {:.info} -> **Note:** Unlike other available AI plugins, the AI MCP plugin is not invoked as part of an AI request. -> Instead, it is registered and executed as a regular plugin, allowing it to capture MCP traffic independently of AI request flow. -> Do not configure the AI MCP plugin together with other `ai-*` plugins on the same service or route. +> **Note:** Unlike other available AI Policies, the AI MCP Policy is not invoked as part of an AI request. +> Instead, it is registered and executed as a regular Policy, allowing it to capture MCP traffic independently of AI request flow. +> Do not configure the AI MCP Policy together with other `ai-*` Policies on the same service or route. The MCP log structure groups traffic by **MCP session ID**, with each session containing zero or more recorded JSON-RPC requests: @@ -573,34 +571,34 @@ rows: description: The size of the JSON-RPC response body, in bytes. - property: "`ai.mcp.audit`" description: | - {% new_in 3.13 %} An array of access control audit entries. Each entry records whether access was allowed or denied for a specific MCP primitive or globally. + An array of access control audit entries. Each entry records whether access was allowed or denied for a specific MCP primitive or globally. - property: "`ai.mcp.audit[].primitive_name`" description: | - {% new_in 3.13 %} The name of the MCP primitive (for example, `list_users`). + The name of the MCP primitive (for example, `list_users`). - property: "`ai.mcp.audit[].primitive`" description: | - {% new_in 3.13 %} The type of MCP primitive (for example, `tool`, `resource`, or `prompt`). + The type of MCP primitive (for example, `tool`, `resource`, or `prompt`). - property: "`ai.mcp.audit[].action`" description: | - {% new_in 3.13 %} The access control decision: `allow` or `deny`. + The access control decision: `allow` or `deny`. - property: "`ai.mcp.audit[].consumer.name`" description: | - {% new_in 3.13 %} The name of the consumer making the request. + The name of the consumer making the request. - property: "`ai.mcp.audit[].consumer.id`" description: | - {% new_in 3.13 %} The UUID of the consumer. + The UUID of the consumer. - property: "`ai.mcp.audit[].consumer.identifier`" description: | - {% new_in 3.13 %} The type of consumer identifier (for example, `consumer_group`). + The type of consumer identifier (for example, `consumer_group`). - property: "`ai.mcp.audit[].scope`" description: | - {% new_in 3.13 %} The scope of the access control check. + The scope of the access control check. {% endtable %} -### AI A2A Proxy logs {% new_in 3.14 %} +### AI A2A Proxy logs -If you're using the [AI A2A Proxy plugin](/plugins/ai-a2a-proxy/), {{site.ai_gateway}} logs include additional fields under the `ai.a2a` object when [`config.logging.log_statistics`](/plugins/ai-a2a-proxy/reference/#schema--config-logging-log-statistics) is enabled. These fields provide observability into Agent-to-Agent (A2A) protocol traffic, including operation names, task lifecycle state, latency, streaming metrics, and optional request/response payloads. +If you create an [AI Policy](/ai-gateway/entities/ai-policy/) using the [AI A2A Proxy Policy](/plugins/ai-a2a-proxy/), {{site.ai_gateway}} logs include additional fields under the `ai.a2a` object when [`config.logging.log_statistics`](/plugins/ai-a2a-proxy/reference/#schema--config-logging-log-statistics) is enabled. These fields provide observability into Agent-to-Agent (A2A) protocol traffic, including operation names, task lifecycle state, latency, streaming metrics, and optional request/response payloads. {% include /plugins/ai-a2a-proxy/log-output-fields.md %} diff --git a/app/ai-gateway/ai-logs.md b/app/ai-gateway/ai-logs.md new file mode 100644 index 00000000000..550817d4394 --- /dev/null +++ b/app/ai-gateway/ai-logs.md @@ -0,0 +1,248 @@ +--- +title: "{{site.ai_gateway}} logs" +content_type: reference +layout: reference +breadcrumbs: + - /ai-gateway/ +products: + - ai-gateway + +tags: + - logging + - monitoring +min_version: + ai-gateway: '2.0' +description: See where {{site.ai_gateway}} logs are located, the different log levels, and how to configure logs and log levels. +search_aliases: + - logging +related_resources: + - text: "Secure {{site.ai_gateway}}" + url: /gateway/security/ + - text: "{{site.ai_gateway}} audit logs" + url: /gateway/audit-logs/ + - text: "{{site.konnect_short_name}} logs" + url: /dedicated-cloud-gateways/konnect-logs/ + - text: "{{site.konnect_short_name}} platform audit logs" + url: /konnect-platform/audit-logs/ + - text: Logging Policies + url: /plugins/?category=logging + - text: Add Correlation IDs to {{site.ai_gateway}} logs + url: /how-to/add-correlation-ids-to-gateway-logs/ + +works_on: + - konnect +--- + +Logging in {{site.ai_gateway}} allows you to see information, warnings, and errors about requests that are proxied by {{site.ai_gateway}}. + +The information in this reference doc helps you understand and modify {{site.ai_gateway}} logs. You can also set Policies with [logging Policies](/plugins/?category=logging) to extend these capabilities by logging additional information or sending logs to another application. + +## Where are {{site.ai_gateway}} logs located? + +By default, you can view {{site.ai_gateway}} logs at `/usr/local/kong/logs/error.log`. If you are running {{site.ai_gateway}} in Docker, you can also view them from your Docker container. + +## Log levels + +By default, logs are set to the recommended `notice` level. If logs are too busy, you can increase the level to something like `warn`. + + +{% table %} +columns: + - title: Level + key: level + - title: Description + key: description +rows: + - level: "`debug`" + description: "Provides debug information about the Policy’s run loop and each individual Policy or other components. This should only be used during debugging. If this is enabled for extended periods of time, it can result in excess disk space consumption." + - level: "`info` and `notice`" + description: "Provides information about normal behavior, most of which can be ignored." + - level: "`warn`" + description: "Logs any abnormal behavior that doesn't result in dropped transactions but requires further investigation." + - level: "`error`" + description: "Used for logging errors that result in a request being dropped. For example, getting a `500` error. The rate of these logs must be monitored." + - level: "`crit`" + description: "Used when {{site.ai_gateway}} is working under critical conditions, affecting several clients. `crit` is the highest severity log level." +{% endtable %} + + +## Configure log levels + +You can change log levels dynamically, without restarting {{site.ai_gateway}}, using the Admin API. Alternatively, you can configure log levels using the `log_level` parameter in the [`kong.conf` file](/gateway/configuration/), but this requires you to [restart {{site.ai_gateway}}](/how-to/restart-kong-gateway-container/). + + +{% table %} +columns: + - title: Use case + key: usecase + - title: How to configure + key: config +rows: + - usecase: "View current log level1" + config: "[`/debug/node/log-level/`](/api/gateway/admin-ee/#/operations/get-debug-node-log-level/)" + - usecase: "Modify the log level for an individual {{site.ai_gateway}} node" + config: "[`/debug/node/log-level/{logLevel}`](/api/gateway/admin-ee/#/operations/get-debug-node-log-level-log_level/)" + - usecase: "Change the log level of the {{site.ai_gateway}} cluster" + config: "[`/debug/cluster/log-level/{loglevel}`](/api/gateway/admin-ee/#/operations/update-debug-cluster-log-level/)" + - usecase: "Keep the log level of new nodes added to the cluster in sync with other nodes in the cluster" + config: | + Change the [`log_level`](/gateway/configuration/#log-level) entry in `kong.conf` to `KONG_LOG_LEVEL`, and start every new node with the `KONG_LOG_LEVEL` env variable set. + - usecase: "Change the log level of all Control Plane {{site.ai_gateway}} nodes" + config: "[`/debug/cluster/control-planes-nodes/log-level/{loglevel}`](/api/gateway/admin-ee/#/operations/create-debug-cluster-control-planes-nodes-log-level)" +{% endtable %} + + +{:.info} +> 1: You can't change the log level of the Data Plane or DB-less nodes. + + +## Find specific client requests in logs + +The `X-Kong-Request-Id` header contains a unique identifier for each client request. You can use this header to match specific requests to their corresponding error logs. + +If {{site.ai_gateway}} returns an error by calling the PDK `kong.response.error`, the request ID will also be included in the response body generated by {{site.ai_gateway}}. In addition, any generated {{site.ai_gateway}} error log contains the same request ID with the format `request_id: xxx`. This can help with debugging because you can search for the header when the debug output is too long to fit in the response header. + +This feature can be customized for upstreams and downstreams using the `headers` and `headers_upstream` configuration options in [`kong.conf`](/gateway/configuration/): + + +{% kong_config_table %} +config: + - name: headers + - name: headers_upstream +{% endkong_config_table %} + + +## Customize what {{site.ai_gateway}} logs + +You may need to customize what {{site.ai_gateway}} logs. For instance, you may want to: +* Protect private information +* Comply with GDPR or other data protection regulations +* Remove instances of a specific piece of data from your logs, such as an email address + +These changes can be made to {{site.ai_gateway}}'s Nginx template and only affect the output of the Nginx access logs. This doesn't have any effect on {{site.ai_gateway}}'s [logging Policies](/plugins/?category=logging). + +Let's look at an example where you want to remove any instances of an email address from your {{site.ai_gateway}} logs. The email addresses may come through in different formats, for example `/servicename/v2/verify/alice@example.com` or `/v3/verify?alice@example.com`. To keep all of these formats from being added to the logs, you need to use a custom Nginx template. + +Make a copy of {{site.ai_gateway}}'s Nginx template, then edit it to add or remove the data you need. The following template shows an example configuration for removing email addresses from logs: + +```nginx +# --------------------- +# custom_nginx.template +# --------------------- + +worker_processes ${{NGINX_WORKER_PROCESSES}}; # can be set by kong.conf +daemon ${{NGINX_DAEMON}}; # can be set by kong.conf + +pid pids/nginx.pid; # this setting is mandatory +error_log stderr ${{LOG_LEVEL}}; # can be set by kong.conf + + + +events { + use epoll; # custom setting + multi_accept on; +} + +http { + + + map $request_uri $keeplog { + ~.+\@.+\..+ 0; + ~/v1/invitation/ 0; + ~/reset/v1/customer/password/token 0; + ~/v2/verify 0; + + default 1; + } + log_format show_everything '$remote_addr - $remote_user [$time_local] ' + '$request_uri $status $body_bytes_sent ' + '"$http_referer" "$http_user_agent"'; + + include 'nginx-kong.conf'; +} +``` + +For this example, we're using the following: + +* `map $request_uri $keeplog`: Maps a new variable called `keeplog`, which is dependent on values appearing in the `$request_uri`. Each line in the example starts with a `~` because this is what tells Nginx to use a regex when evaluating the line. This example looks for the following: + - The first line uses a regex to look for any email address in the `x@y.z` format + - The second line looks for any part of the URI that contains `/servicename/v2/verify` + - The third line looks at any part of the URI that contains `/v3/verify` + + Because all of these patterns have a value of something other than `0`, if a request has any of those elements, it will not be added to the log. +* `log_format`: Sets the log format for what {{site.ai_gateway}} keeps in the logs. The contents of the log can be customized for your needs. For the purpose of this example, you can assign the new logs with the name `show_everything` and set everything to the {{site.ai_gateway}} default standards. To see the full list of options, refer to the [Nginx core module variables reference](https://nginx.org/en/docs/http/ngx_http_core_module.html#variables). + +Once you've adjusted the Nginx template for your environment, you need to tell {{site.ai_gateway}} to use the newly created log, `show_everything`. + +To do this, alter the {{site.ai_gateway}} variable `proxy_access_log` by either editing `etc/kong/kong.conf` or using the environmental variable `KONG_PROXY_ACCESS_LOG` adjust the default location: + +```sh +proxy_access_log=logs/access.log show_everything if=$keeplog +``` + +Restart {{site.ai_gateway}} to apply changes with the `kong restart` command. + +Now, any request made with an email address in it will no longer be logged. + +## {{site.ai_gateway}} logs + +{{site.ai_gateway}} collects logs for the [{{site.ai_gateway}} Policies](/plugins/?category=ai). This allows you to aggregate AI usage analytics across various providers. + +Each log entry includes the following details: + + +{% table %} +columns: + - title: Property + key: property + - title: Description + key: description +rows: + - property: "`ai.$PLUGIN_NAME.payload.request`" + description: The request payload. + - property: "`ai.$PLUGIN_NAME.payload.response`" + description: The response payload. + - property: "`ai.$PLUGIN_NAME.usage.prompt_token`" + description: The number of tokens used for prompting. + - property: "`ai.$PLUGIN_NAME.usage.completion_token`" + description: The number of tokens used for completion. + - property: "`ai.$PLUGIN_NAME.usage.total_tokens`" + description: The total number of tokens used. + - property: "`ai.$PLUGIN_NAME.usage.cost`" + description: The total cost of the request (input and output cost). + - property: "`ai.$PLUGIN_NAME.usage.time_per_token`" + description: | + The average time to generate an output token, in milliseconds. + - property: "`ai.$PLUGIN_NAME.meta.request_model`" + description: The model used for the AI request. + - property: "`ai.$PLUGIN_NAME.meta.provider_name`" + description: The name of the AI service provider. + - property: "`ai.$PLUGIN_NAME.meta.response_model`" + description: The model used for the AI response. + - property: "`ai.$PLUGIN_NAME.meta.plugin_id`" + description: The unique identifier of the Policy. + - property: "`ai.$PLUGIN_NAME.meta.llm_latency`" + description: | + The time, in milliseconds, it took the LLM provider to generate the full response. + - property: "`ai.$PLUGIN_NAME.cache.cache_status`" + description: | + The cache status. This can be `Hit`, `Miss`, `Bypass` or `Refresh`. + - property: "`ai.$PLUGIN_NAME.cache.fetch_latency`" + description: | + The time, in milliseconds, it took to return a cache response. + - property: "`ai.$PLUGIN_NAME.cache.embeddings_provider`" + description: | + For semantic caching, the provider used to generate the embeddings. + - property: "`ai.$PLUGIN_NAME.cache.embeddings_model`" + description: | + For semantic caching, the model used to generate the embeddings. + - property: "`ai.$PLUGIN_NAME.cache.embeddings_latency`" + description: | + For semantic caching, the time taken to generate the embeddings. +{% endtable %} + + + diff --git a/app/ai-gateway/ai-otel-metrics.md b/app/ai-gateway/ai-otel-metrics.md index 091c997b7fa..186fe2b13a6 100644 --- a/app/ai-gateway/ai-otel-metrics.md +++ b/app/ai-gateway/ai-otel-metrics.md @@ -5,7 +5,6 @@ layout: reference products: - ai-gateway - - gateway breadcrumbs: - /ai-gateway/ @@ -18,11 +17,9 @@ tags: plugins: - opentelemetry - - ai-proxy - - ai-proxy-advanced min_version: - gateway: '3.14' + ai-gateway: '2.0' tech_preview: true toc_depth: 2 @@ -34,13 +31,9 @@ related_resources: url: /ai-gateway/llm-open-telemetry/ - text: "Monitor AI LLM metrics (Prometheus)" url: /ai-gateway/monitor-ai-llm-metrics/ - - text: "Proxy A2A agents through {{site.ai_gateway}}" - url: /how-to/proxy-a2a-agents/ - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: "{{site.ai_gateway}} plugins" - url: /plugins/?category=ai - - text: OpenTelemetry plugin + - text: OpenTelemetry Policy url: /plugins/opentelemetry/ - text: Full OpenTelemetry metrics reference url: /gateway/otel-metrics/ @@ -48,15 +41,12 @@ related_resources: url: /gateway/tracing/ works_on: - - on-prem - konnect --- -{% new_in 3.14 %} {{site.ai_gateway}} can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through the [OpenTelemetry plugin](/plugins/opentelemetry/). These metrics are aggregated time-series data points (counters, histograms) pushed to a configured OTLP metrics endpoint on a regular interval. They are separate from the per-request [Gen AI span attributes](/ai-gateway/llm-open-telemetry/) emitted on traces. +{{site.ai_gateway}} can export OpenTelemetry (OTLP) metrics for generative AI, MCP, and A2A traffic through an [OpenTelemetry AI Policy](/plugins/opentelemetry/). These metrics are aggregated time-series data points (counters, histograms) pushed to a configured OTLP metrics endpoint on a regular interval. They are separate from the per-request [Gen AI span attributes](/ai-gateway/llm-open-telemetry/) emitted on traces. -For a step-by-step setup using an OpenTelemetry Collector, see [Collect metrics, logs, and traces with the OpenTelemetry plugin](/how-to/collect-metrics-logs-and-traces-with-opentelemetry/). To visualize Gen AI traces in Jaeger, see [Set up Jaeger with Gen AI OpenTelemetry](/how-to/set-up-jaeger-with-gen-ai-otel/). - -Use these metrics to: +You can use these metrics to: * Track LLM request latency and upstream provider processing time * Monitor token consumption across providers, models, and consumers @@ -67,409 +57,65 @@ Use these metrics to: ## Prerequisites -To collect AI OTel metrics, enable the following settings: +To collect AI OTLP metrics, enable the following settings: {% table %} columns: - title: Setting key: setting - - title: Plugin - key: plugin + - title: Policy + key: policy - title: Required for key: required_for rows: - setting: "`config.metrics.enable_ai_metrics`: `true`" - plugin: "[OpenTelemetry](/plugins/opentelemetry/reference/)" + policy: "[OpenTelemetry](/plugins/opentelemetry/reference/)" required_for: "All AI metrics" - setting: "`config.metrics.endpoint`" - plugin: "[OpenTelemetry](/plugins/opentelemetry/reference/)" + policy: "[OpenTelemetry](/plugins/opentelemetry/reference/)" required_for: "All AI metrics (set to a valid OTLP-compatible metrics endpoint)" - setting: "`config.logging.log_statistics`: `true`" - plugin: "[AI Proxy](/plugins/ai-proxy/reference/) or [AI Proxy Advanced](/plugins/ai-proxy-advanced/reference/)" - required_for: "[Gen AI metrics](#gen-ai-metrics-otel-semantic-conventions)" + policy: "[AI Proxy](/plugins/ai-proxy/reference/) or [AI Proxy Advanced](/plugins/ai-proxy-advanced/reference/)" + required_for: "[Gen AI metrics](#gen-ai-metrics-otlp-semantic-conventions)" - setting: "`config.logging.log_statistics`: `true`" - plugin: "[AI MCP Proxy](/plugins/ai-mcp-proxy/reference/)" + policy: "[AI MCP Proxy](/plugins/ai-mcp-proxy/reference/)" required_for: "[MCP metrics](#mcp-metrics)" - setting: "`config.logging.log_statistics`: `true`" - plugin: "[AI A2A Proxy](/plugins/ai-a2a-proxy/reference/)" + policy: "[AI A2A Proxy](/plugins/ai-a2a-proxy/reference/)" required_for: "[A2A metrics](#a2a-metrics)" {% endtable %} Some metrics have additional requirements: -* `gen_ai.server.request.duration` and `mcp.client.operation.duration` require `config.metrics.enable_latency_metrics` set to `true` in the [OpenTelemetry plugin](/plugins/opentelemetry/reference/). -* The `error.type` attribute on duration metrics requires `config.metrics.enable_request_metrics` set to `true` in the [OpenTelemetry plugin](/plugins/opentelemetry/reference/). +* `gen_ai.server.request.duration` and `mcp.client.operation.duration` require `config.metrics.enable_latency_metrics` set to `true` in the [OpenTelemetry AI Policy](/plugins/opentelemetry/reference/). +* The `error.type` attribute on duration metrics requires `config.metrics.enable_request_metrics` set to `true` in the [OpenTelemetry AI Policy](/plugins/opentelemetry/reference/). -## Gen AI metrics (OTel semantic conventions) +## Gen AI metrics (OTLP semantic conventions) These metrics follow the [OpenTelemetry Gen AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/). They capture request duration, upstream latency, token usage, and streaming performance. ### Metric reference -{% include plugins/otel/metric_tables.md metric_prefixes="gen_ai." %} + +{% include md/ai-gateway/v2/policies/metric_tables.md metric_prefixes="gen_ai." %} ## Kong Gen AI metrics These metrics use the `kong.gen_ai.*` namespace and capture Kong-specific AI observability data, including cost tracking, cache and RAG latency, and AWS Guardrails processing time. -### kong.gen_ai.llm.cost - -Cost of AI requests. To populate this metric, define `model.options.input_cost` and `model.options.output_cost` in the [AI Proxy](/plugins/ai-proxy/reference/#schema--config-model-options-input-cost) or [AI Proxy Advanced](/plugins/ai-proxy-advanced/reference/#schema--config-targets-model-options-input-cost) plugin configuration. - -* **Type**: Counter -* **Unit**: `{cost}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`gen_ai.provider.name`" - desc: "Name of the Gen AI provider." - - attr: "`gen_ai.request.model`" - desc: "Model name targeted by the request." - - attr: "`gen_ai.response.model`" - desc: "Model name reported by the provider in the response." - - attr: "`gen_ai.operation.name`" - desc: "Operation requested, such as `chat` or `embeddings`." - - attr: "`kong.gen_ai.cache.status`" - desc: "Cache status: `hit` or empty if not cached." - - attr: "`kong.gen_ai.vector_db`" - desc: "Vector database used for caching, such as `redis`." - - attr: "`kong.gen_ai.embeddings.provider`" - desc: "Embeddings provider used for caching." - - attr: "`kong.gen_ai.embeddings.model`" - desc: "Embeddings model used for caching." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.auth.consumer.name`" - desc: "Name of the authenticated Consumer." - - attr: "`kong.gen_ai.request.mode`" - desc: "Request mode: `oneshot`, `stream`, or `realtime`." -{% endtable %} - - -### kong.gen_ai.cache.fetch.latency - -Time to fetch a response from the semantic cache. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.llm.cost`](#konggen_aillmcost). - -### kong.gen_ai.cache.embeddings.latency +To populate `kong.gen_ai.llm.cost`, define `model.options.input_cost` and `model.options.output_cost` in your model configuration. -Time to generate embeddings during cache operations. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.llm.cost`](#konggen_aillmcost). - -### kong.gen_ai.rag.fetch.latency - -Time to fetch data from a RAG (Retrieval-Augmented Generation) source. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.llm.cost`](#konggen_aillmcost). - -### kong.gen_ai.rag.embeddings.latency - -Time to generate embeddings for RAG operations. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.llm.cost`](#konggen_aillmcost). - -### kong.gen_ai.aws.guardrails.latency - -Time for AWS Guardrails to process a request. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.gen_ai.aws.guardrails.id`" - desc: "ID of the AWS Guardrails configuration." - - attr: "`kong.gen_ai.aws.guardrails.version`" - desc: "Version of the AWS Guardrails configuration." - - attr: "`kong.gen_ai.aws.guardrails.mode`" - desc: "Mode of the AWS Guardrails evaluation." - - attr: "`kong.gen_ai.aws.guardrails.region`" - desc: "AWS region of the Guardrails service." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.auth.consumer.name`" - desc: "Name of the authenticated Consumer." -{% endtable %} - +{% include md/ai-gateway/v2/policies/metric_tables.md metric_prefixes="kong.gen_ai." %} ## MCP metrics These metrics provide observability into MCP (Model Context Protocol) server interactions, including latency, response sizes, errors, and ACL decisions. -### mcp.client.operation.duration - -Duration of the MCP request as observed by the sender. Only available when the [AI MCP Proxy plugin](/plugins/ai-mcp-proxy/) is in passthrough-listener mode (the upstream is an MCP server). Requires `enable_latency_metrics` set to `true`. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`mcp.method.name`" - desc: "MCP method name, such as `tools/call`." - - attr: "`gen_ai.tool.name`" - desc: "Name of the tool invoked." - - attr: "`error.type`" - desc: "JSON-RPC error code, if the request failed." - - attr: "`gen_ai.operation.name`" - desc: "Operation name, such as `execute_tool` for `tools/call`." -{% endtable %} - - -### mcp.server.operation.duration - -Duration of the MCP request as observed by the receiver. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`mcp.client.operation.duration`](#mcpclientoperationduration). - -### kong.gen_ai.mcp.response.size - -Size of the MCP response body. - -* **Type**: Histogram -* **Unit**: `By` (bytes) - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`mcp.method.name`" - desc: "MCP method name, such as `tools/call`." - - attr: "`gen_ai.tool.name`" - desc: "Name of the tool invoked." -{% endtable %} - - -### kong.gen_ai.mcp.request.error.count - -Number of MCP request errors. - -* **Type**: Counter -* **Unit**: `{error}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`mcp.method.name`" - desc: "MCP method name, such as `tools/call`." - - attr: "`gen_ai.tool.name`" - desc: "Name of the tool invoked." - - attr: "`error.type`" - desc: "JSON-RPC error code." -{% endtable %} - - -### kong.gen_ai.mcp.acl.allowed - -Number of MCP requests allowed by ACL rules. - -* **Type**: Counter -* **Unit**: `{request}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.gen_ai.mcp.primitive`" - desc: "MCP primitive type, such as `tool`." - - attr: "`kong.gen_ai.mcp.primitive_name`" - desc: "Name of the MCP primitive." -{% endtable %} - - -### kong.gen_ai.mcp.acl.denied - -Number of MCP requests denied by ACL rules. - -* **Type**: Counter -* **Unit**: `{request}` - -**Attributes:** Same as [`kong.gen_ai.mcp.acl.allowed`](#konggen_aimcpaclallowed). +{% include md/ai-gateway/v2/policies/metric_tables.md metric_prefixes="mcp.,kong.gen_ai.mcp." %} ## A2A metrics These metrics provide observability into [A2A (Agent-to-Agent)](/plugins/ai-a2a-proxy/) traffic, including request volume, latency, response sizes, and task state transitions. -### kong.gen_ai.a2a.request.count - -Total number of A2A requests. - -* **Type**: Counter -* **Unit**: `{request}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.gen_ai.a2a.method`" - desc: "A2A method name." - - attr: "`kong.gen_ai.a2a.binding`" - desc: "A2A binding type." -{% endtable %} - - -### kong.gen_ai.a2a.request.duration - -Duration of an A2A request. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.a2a.request.count`](#konggen_aia2arequestcount). - -### kong.gen_ai.a2a.response.size - -Size of the A2A response body. - -* **Type**: Histogram -* **Unit**: `By` (bytes) - -**Attributes:** Same as [`kong.gen_ai.a2a.request.count`](#konggen_aia2arequestcount). - -### kong.gen_ai.a2a.ttfb - -Time to first byte for A2A streaming responses. - -* **Type**: Histogram -* **Unit**: `s` (seconds) - -**Attributes:** Same as [`kong.gen_ai.a2a.request.count`](#konggen_aia2arequestcount). - -### kong.gen_ai.a2a.request.error.count - -Number of A2A request errors. - -* **Type**: Counter -* **Unit**: `{error}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.gen_ai.a2a.method`" - desc: "A2A method name." - - attr: "`kong.gen_ai.a2a.binding`" - desc: "A2A binding type." - - attr: "`kong.gen_ai.a2a.error.type`" - desc: "Type of the A2A error." -{% endtable %} - - -### kong.gen_ai.a2a.task.state.count - -Number of A2A task state transitions. - -* **Type**: Counter -* **Unit**: `{state}` - - -{% table %} -columns: - - title: Attribute - key: attr - - title: Description - key: desc -rows: - - attr: "`kong.service.name`" - desc: "Name of the Gateway Service." - - attr: "`kong.route.name`" - desc: "Name of the Route." - - attr: "`kong.workspace.name`" - desc: "Name of the Workspace." - - attr: "`kong.gen_ai.a2a.task.state`" - desc: "Task state, such as `completed`, `failed`, or `in_progress`." -{% endtable %} - +{% include md/ai-gateway/v2/policies/metric_tables.md metric_prefixes="kong.gen_ai.a2a." %} diff --git a/app/ai-gateway/llm-open-telemetry.md b/app/ai-gateway/llm-open-telemetry.md index f67cc37c9ed..b7b166ffd46 100644 --- a/app/ai-gateway/llm-open-telemetry.md +++ b/app/ai-gateway/llm-open-telemetry.md @@ -7,7 +7,6 @@ toc_depth: 4 products: - ai-gateway - - gateway breadcrumbs: - /ai-gateway/ @@ -19,11 +18,9 @@ tags: plugins: - opentelemetry - - ai-proxy - - ai-proxy-advanced min_version: - gateway: '3.13' + ai-gateway: '2.0' tech_preview: true @@ -34,29 +31,22 @@ related_resources: url: /ai-gateway/ai-otel-metrics/ - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: "{{site.ai_gateway}} plugins" - url: /plugins/?category=ai - - text: OpenTelemetry plugin + - text: OpenTelemetry Policy url: /plugins/opentelemetry/ - - text: Zipkin plugin + - text: Zipkin Policy url: /plugins/zipkin/ - text: "{{site.base_gateway}} tracing guide" url: /gateway/tracing/ - - text: Set up Jaeger with Gen AI OpenTelemetry - url: /how-to/set-up-jaeger-with-gen-ai-otel/ - - text: Validate Gen AI tool calls with Jaeger and OpenTelemetry - url: /how-to/set-up-jaeger-with-gen-ai-otel-for-tool-calls/ works_on: - - on-prem - konnect --- -{% new_in 3.13 %} {{site.ai_gateway}} supports [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/#genai-attributes) instrumentation for generative AI traffic. When the OpenTelemetry (OTEL) plugin is enabled in {{site.ai_gateway}}, a set of **Gen AI-specific attributes** are emitted on tracing spans. These attributes complement the core tracing instrumentations described in the [{{site.base_gateway}} tracing guide](/gateway/tracing), giving insight into the Gen AI request lifecycle (inputs, model, and outputs), usage, and tool/agent interactions. +{{site.ai_gateway}} supports [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/#genai-attributes) instrumentation for generative AI traffic. When an OpenTelemetry (OTEL) Policy is enabled in {{site.ai_gateway}}, a set of **Gen AI-specific attributes** are emitted on tracing spans. These attributes provide insight into the Gen AI request lifecycle (inputs, model, and outputs), usage, and tool or agent interactions. -{% new_in 3.14 %} [A2A agent traffic](#a2a-span-attributes) is also instrumented via the [AI A2A Proxy plugin](/plugins/ai-a2a-proxy/). +You can also capture [A2A agent traffic](#a2a-span-attributes) by enabling statistics logging on [AI Agents](/ai-gateway/entities/ai-agent/#logging-and-observability). -You can export these attributes via a supported backend such as [Jaeger](/how-to/set-up-jaeger-with-otel/) configured through Kong's [OpenTelemetry plugin](/plugins/opentelemetry) or the [Zipkin plugin](/plugins/zipkin) to: +You can export these attributes via a supported backend to: * Inspect which model or provider handled a request * Track conversation/session identifiers across requests @@ -65,19 +55,17 @@ You can export these attributes via a supported backend such as [Jaeger](/how-to * Measure tool-call behavior (which tools were invoked, and their metadata) * Monitor token usage (input vs. output) for cost or performance analysis -The span data is sent to the configured OTEL endpoint through the existing tracing plugins. Use the OpenTelemetry plugin or Zipkin plugin to export these spans to backends such as Jaeger. +The span data is sent to the configured OTEL endpoint through the [Kong tracing](/gateway/tracing/). Use a Policy configured with OpenTelemetry or Zipkin to export these spans to backends such as Jaeger. {:.info} > This page covers **span attributes** (per-request tracing data). {{site.ai_gateway}} also supports **OTLP metrics** (aggregated counters and histograms for latency, token usage, cost, and error rates). See the [Gen AI OpenTelemetry metrics reference](/ai-gateway/ai-otel-metrics/) for details. -{% include plugins/otel/collecting-otel-data.md %} - {:.warning} > Some Gen AI span attributes can include sensitive request or response payload data. In particular, `gen_ai.input.messages` and `gen_ai.output.messages` may contain prompts, model outputs, PII, secrets, or credentials. Review your tracing, retention, access-control, and redaction requirements before enabling or exporting payload-related tracing data. ## Span attribute reference -{% include plugins/otel/span_attribute_tables.md %} +{% include md/ai-gateway/v2/policies/span_attribute_tables.md %} diff --git a/app/ai-gateway/semantic-similarity.md b/app/ai-gateway/semantic-similarity.md index 4209c67b1d5..43afbb18877 100644 --- a/app/ai-gateway/semantic-similarity.md +++ b/app/ai-gateway/semantic-similarity.md @@ -22,7 +22,7 @@ min_version: related_resources: - text: "{{site.ai_gateway}}" url: /ai-gateway/ - - text: Policy entity + - text: AI Policy entity url: /ai-gateway/entities/ai-policy/ - text: "{{site.ai_gateway}} Model entity" url: /ai-gateway/entities/ai-model/ @@ -45,15 +45,15 @@ For example, in the figure 1, “king” and “emperor” are semantically more ## Semantic similarity in {{site.ai_gateway}} -Based on meaning rather than exact matches, {{site.ai_gateway}} can perform intelligent request routing, caching, and content filtering using semantic similarity queries. A [Model](/ai-gateway/entities/ai-model/) can leverage semantic similarity in two ways: +Based on meaning rather than exact matches, {{site.ai_gateway}} can perform intelligent request routing, caching, and content filtering using semantic similarity queries. An [AI Model](/ai-gateway/entities/ai-model/) can leverage semantic similarity in two ways: 1. **Semantic load balancing**: Route requests to upstream providers based on how semantically similar the prompt is to each provider's capabilities, using the `semantic` load balancing algorithm. -2. **Semantic Policies**: Attach Policies like AI Semantic Cache or AI Semantic Prompt Guard to add similarity-based caching, retrieval-augmented generation (RAG), and guardrails. +2. **Semantic Policies**: Attach AI Policies like AI Semantic Cache or AI Semantic Prompt Guard to add similarity-based caching, retrieval-augmented generation (RAG), and guardrails. ### Vector databases To store and compare embeddings efficiently, {{site.ai_gateway}} semantic features rely on vector databases. These specialized datastores index high-dimensional embeddings and enable **fast similarity search** based on distance metrics like cosine similarity or Euclidean distance. -A Model Entity’s [semantic load balancer](/ai-gateway/entities/ai-model/#algorithms) stores vector representations of each target model’s semantic description at configuration time, and uses the vector database to compare incoming prompts against those stored vectors. +An AI Model entity’s [semantic load balancer](/ai-gateway/entities/ai-model/#algorithms) stores vector representations of each target model’s semantic description at configuration time, and uses the vector database to compare incoming prompts against those stored vectors. Semantic policies also use vector databases to perform similarity searches at request time. The selected database stores the embeddings generated by the Model or Policies (either at config time or runtime), and determines the accuracy and performance of semantic operations. @@ -266,7 +266,7 @@ The threshold defines how permissive the matching is. **Higher threshold values * For **Euclidean distance**, the threshold is normalized to a 0–1 range and sets the maximum allowable distance between embedding vectors. A value of `0` requires exact matches (zero distance). A value of `1` permits the broadest possible matches. Typical configurations use `0.1–0.2` for strict matching and `0.5–0.8` for broader matching. -In both cases, if the [{{site.base_gateway}} logs](/gateway/logs/) indicate "no target can be found under threshold X," increase the threshold value to allow more matches. +In both cases, if the [{{site.ai_gateway}} logs](/ai-gateway/ai-logs/) indicate "no target can be found under threshold X," increase the threshold value to allow more matches. The optimal threshold depends on the selected distance metric, the embedding model's dimensionality, and the variation in your data. Tuning may be required for best results.