diff --git a/docs/developer-lightspeed-guide/master.adoc b/docs/developer-lightspeed-guide/master.adoc index 6da31f357f2..a289d06811e 100644 --- a/docs/developer-lightspeed-guide/master.adoc +++ b/docs/developer-lightspeed-guide/master.adoc @@ -23,6 +23,9 @@ include::topics/developer-lightspeed/assembly_run-dev-lightspeed-analysis.adoc[l include::topics/developer-lightspeed/con_developer-lightspeed-logs.adoc[leveloffset=+1] +include::topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc[leveloffset=+1] + include::topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc[leveloffset=+1] + :!mta-developer-lightspeed: diff --git a/docs/topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc b/docs/topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc new file mode 100644 index 00000000000..93258d36bee --- /dev/null +++ b/docs/topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc @@ -0,0 +1,15 @@ +// Module included in the following assemblies: +// +// * docs/developer-lightspeed-guide/master.adoc + +:_mod-docs-content-type: ASSEMBLY + +[id="assembly-llm-proxy-interaction-monitoring_{context}"] += LLM proxy interaction monitoring + +[role="_abstract"] +The {mta-dl-plugin} large language model (LLM) proxy acts as a single, administrator-controlled access point to external LLM providers. You can monitor its data handling and logging behaviors to meet artificial intelligence (AI) governance requirements. + +include::con_llm-proxy-data-handling.adoc[leveloffset=+1] + +include::proc_access-llm-proxy-logs.adoc[leveloffset=+1] diff --git a/docs/topics/developer-lightspeed/con_llm-proxy-data-handling.adoc b/docs/topics/developer-lightspeed/con_llm-proxy-data-handling.adoc new file mode 100644 index 00000000000..d722e34af99 --- /dev/null +++ b/docs/topics/developer-lightspeed/con_llm-proxy-data-handling.adoc @@ -0,0 +1,41 @@ +// Module included in the following assemblies: +// +// * docs/developer-lightspeed-guide/master.adoc + +:_mod-docs-content-type: CONCEPT + +[id="llm-proxy-data-handling_{context}"] += LLM proxy request and data handling + +[role="_abstract"] +The large language model (LLM) proxy is built on llama-stack and runs as a dedicated pod within the {ProductShortName} namespace. + +The proxy provides the following controls: + +Credential isolation:: LLM provider application programming interface (API) keys are stored in a Kubernetes secret and never distributed to client applications. Only the proxy pod reads these credentials. + +JWT-based authentication:: When authentication is enabled, the proxy validates every incoming JSON web token (JWT) request against the Hub OpenID Connect (OIDC) provider by using its JWKS endpoint: ++ +[subs="+quotes"] +---- +https://____/realms/____/protocol/openid-connect/certs +---- ++ +where: ++ +``:: Specifies the name of the OIDC provider. +``:: Specifies the name of the realm. ++ +Requests that fail validation are rejected with a `401` or `403` response before they reach the LLM provider. + +Header redaction:: Authorization headers and other sensitive fields are redacted from proxy logs. + +Data persistence:: LLM interactions are stored in the llama-stack PostgreSQL database in the following tables: ++ +* `llm_proxy_inference_store`: Individual inference records. +* `llm_proxy_conversations`: Conversation context records. + +[NOTE] +==== +As of {ProductShortName} 8.2, OIDC user identity is terminated at the Hub and is not passed to or stored by the proxy. +==== diff --git a/docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc b/docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc new file mode 100644 index 00000000000..647ba2bd232 --- /dev/null +++ b/docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc @@ -0,0 +1,56 @@ +// Module included in the following assemblies: +// +// * docs/developer-lightspeed-guide/master.adoc + +:_mod-docs-content-type: PROCEDURE + +[id="llm-proxy-log-access_{context}"] += Access LLM proxy pod logs + +[role="_abstract"] +To diagnose connectivity errors between the proxy and the large language model (LLM) provider, use the `oc logs` command. The proxy writes standard server output to stdout. + +Current proxy logs are unstructured uvicorn server output. Each entry contains the HTTP method, request path, and status code, or a Python exception traceback for error conditions. They do not contain structured per-transaction records or user identity information. + +.Prerequisites +* You have installed the {ocp-short} {CLIName} (`oc`). +* You are logged in to the cluster with the `cluster-admin` role or an equivalent role that grants read access to pods in the {ProductShortName} namespace. + +.Procedure + +. Identify the proxy pod name: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get pods -n ____ | grep proxy +---- ++ +where: ++ +``:: Specifies the name of the {ProductShortName} namespace. ++ +The command returns output similar to the following example: ++ +[source,terminal] +---- +mta-ui-mta-proxy-7d6f9b8c4-xk2pq 1/1 Running 0 2d +---- + +. Stream the current log output from the proxy pod: ++ +[source,terminal,subs="+quotes"] +---- +$ oc logs -f ____ -n ____ +---- ++ +where: ++ +``:: Specifies the name of the proxy pod. + +. To retrieve logs from a previous (terminated) proxy pod, add the `--previous` flag to the command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc logs --previous ____ -n ____ +---- +