Skip to content

MTA-7134-LLM-Proxy-Transaction-Logging-Schema-Audit-Trail-Integration #379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anarnold97 merged 13 commits into from
Jul 31, 2026
Merged

MTA-7134-LLM-Proxy-Transaction-Logging-Schema-Audit-Trail-Integration #379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
3731a39
MTA-7134-LLM-Proxy-Transaction-Logging-Schema-Audit-Trail-Integration
anarnold97 Jul 2, 2026
701a92e
Update docs/topics/developer-lightspeed/proc_auditing-llm-proxy-trans…
anarnold97 Jul 20, 2026
5e030ef
Update docs/topics/developer-lightspeed/proc_auditing-llm-proxy-trans…
anarnold97 Jul 20, 2026
c683bab
Update docs/topics/developer-lightspeed/proc_auditing-llm-proxy-trans…
anarnold97 Jul 20, 2026
684a8d6
Modularize
anarnold97 Jul 20, 2026
5c51b9d
removing patch
anarnold97 Jul 20, 2026
c699e7b
Update assembly_llm-proxy-interaction-monitoring.adoc
anarnold97 Jul 20, 2026
b78c040
Update con_llm-proxy-data-handling.adoc
anarnold97 Jul 20, 2026
a911fa9
Update proc_access-llm-proxy-logs.adoc
anarnold97 Jul 20, 2026
3495b4e
Update docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc
anarnold97 Jul 23, 2026
072ac2b
Update docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc
anarnold97 Jul 23, 2026
a5681b3
Update docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc
anarnold97 Jul 23, 2026
d3e316d
Merge branch 'main' into MTA-7134-LLM-Proxy-Transaction-Logging-Schem…
anarnold97 Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/developer-lightspeed-guide/master.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ include::topics/developer-lightspeed/assembly_run-dev-lightspeed-analysis.adoc[l

include::topics/developer-lightspeed/con_developer-lightspeed-logs.adoc[leveloffset=+1]

include::topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc[leveloffset=+1]

include::topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc[leveloffset=+1]


:!mta-developer-lightspeed:
15 changes: 15 additions & 0 deletions docs/topics/developer-lightspeed/assembly_llm-proxy-interaction-monitoring.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Module included in the following assemblies:
//
// * docs/developer-lightspeed-guide/master.adoc

:_mod-docs-content-type: ASSEMBLY

[id="assembly-llm-proxy-interaction-monitoring_{context}"]
= LLM proxy interaction monitoring

[role="_abstract"]
The {mta-dl-plugin} large language model (LLM) proxy acts as a single, administrator-controlled access point to external LLM providers. You can monitor its data handling and logging behaviors to meet artificial intelligence (AI) governance requirements.

include::con_llm-proxy-data-handling.adoc[leveloffset=+1]

include::proc_access-llm-proxy-logs.adoc[leveloffset=+1]
41 changes: 41 additions & 0 deletions docs/topics/developer-lightspeed/con_llm-proxy-data-handling.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Module included in the following assemblies:
//
// * docs/developer-lightspeed-guide/master.adoc

:_mod-docs-content-type: CONCEPT

[id="llm-proxy-data-handling_{context}"]
= LLM proxy request and data handling

[role="_abstract"]
The large language model (LLM) proxy is built on llama-stack and runs as a dedicated pod within the {ProductShortName} namespace.
Comment thread
anarnold97 marked this conversation as resolved.

The proxy provides the following controls:

Credential isolation:: LLM provider application programming interface (API) keys are stored in a Kubernetes secret and never distributed to client applications. Only the proxy pod reads these credentials.

JWT-based authentication:: When authentication is enabled, the proxy validates every incoming JSON web token (JWT) request against the Hub OpenID Connect (OIDC) provider by using its JWKS endpoint:
+
[subs="+quotes"]
----
https://__<oidc_provider>__/realms/__<realm>__/protocol/openid-connect/certs
----
+
where:
+
`<oidc_provider>`:: Specifies the name of the OIDC provider.
`<realm>`:: Specifies the name of the realm.
+
Requests that fail validation are rejected with a `401` or `403` response before they reach the LLM provider.

Header redaction:: Authorization headers and other sensitive fields are redacted from proxy logs.

Data persistence:: LLM interactions are stored in the llama-stack PostgreSQL database in the following tables:
+
* `llm_proxy_inference_store`: Individual inference records.
* `llm_proxy_conversations`: Conversation context records.

[NOTE]
====
As of {ProductShortName} 8.2, OIDC user identity is terminated at the Hub and is not passed to or stored by the proxy.
====
Comment thread
anarnold97 marked this conversation as resolved.
56 changes: 56 additions & 0 deletions docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Module included in the following assemblies:
//
// * docs/developer-lightspeed-guide/master.adoc

:_mod-docs-content-type: PROCEDURE

[id="llm-proxy-log-access_{context}"]
= Access LLM proxy pod logs

[role="_abstract"]
To diagnose connectivity errors between the proxy and the large language model (LLM) provider, use the `oc logs` command. The proxy writes standard server output to stdout.

Current proxy logs are unstructured uvicorn server output. Each entry contains the HTTP method, request path, and status code, or a Python exception traceback for error conditions. They do not contain structured per-transaction records or user identity information.

.Prerequisites
* You have installed the {ocp-short} {CLIName} (`oc`).
* You are logged in to the cluster with the `cluster-admin` role or an equivalent role that grants read access to pods in the {ProductShortName} namespace.
Comment on lines +15 to +17

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Repository files matching target:"
fd -a 'proc_access-llm-proxy-logs\.adoc$' . || true

for f in $(fd 'proc_access-llm-proxy-logs\.adoc$' .); do
  echo "--- $f ---"
  wc -l "$f"
  cat -n "$f"
done

echo "--- Search for related procedure references/commands ---"
rg -n "oc logs|previous|cluster-admin|pods/log|read access to pods|logs" docs/topics/developer-lightspeed .github docs 2>/dev/null | head -n 200 || true

Repository: migtools/mta-documentation

Length of output: 30043

🌐 Web query:

OpenShift oc logs RBAC pods/log subresource list pods log pod access documentation

💡 Result:

To access pod logs in OpenShift using the oc logs command, a user or service account must have specific Role-Based Access Control (RBAC) permissions. Granting permission to view pods (e.g., get or list on the pods resource) does not automatically grant the ability to view logs [1][2][3]. Access to pod logs requires explicit authorization for the pods/log subresource [1][4]. RBAC Configuration Requirements: To authorize log access, define a Role or ClusterRole that includes the pods/log resource within the core API group [1][2]. Example Role Configuration: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pod-log-reader namespace: rules: - apiGroups: [""] resources: ["pods/log"] verbs: ["get", "list"] In this configuration: - apiGroups: [""] refers to the core Kubernetes API group [2]. - resources: ["pods/log"] specifies the subresource required for oc logs [1]. - verbs: ["get", "list"] are the necessary actions to retrieve log data [1][2][3]. Key Considerations: 1. Subresource Specificity: The pods/log subresource must be defined separately from the standard pods resource [2][3]. If you want a user to be able to see the pod list and read the logs, both pods and pods/log must be included in the rules [1][4]. 2. Testing Permissions: You can verify if a user or service account has the necessary permissions by using the following command [3]: oc auth can-i get pods/log --as= --namespace= 3. Least Privilege: Only grant these permissions to users or service accounts that specifically require access to log data, as logs may contain sensitive information [5]. For further details on managing RBAC in OpenShift, refer to the official Red Hat OpenShift authentication and authorization documentation [6].

Citations:

Document the minimum RBAC permissions.

Replace “read access to pods” with the namespace-scoped permissions required by the commands: access to list pods plus pods/log read access. Recommend least-privilege roles instead of cluster-admin.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc` around
lines 15 - 17, Update the Prerequisites entry to document the minimum
namespace-scoped RBAC needed by the commands: permission to list pods and read
the pods/log subresource in the {ProductShortName} namespace. Remove the
cluster-admin recommendation and direct users toward least-privilege roles
granting only these permissions.

Source: MCP tools


.Procedure

. Identify the proxy pod name:
+
[source,terminal,subs="+quotes"]
----
$ oc get pods -n __<mta_namespace>__ | grep proxy
----
+
where:
+
`<mta_namespace>`:: Specifies the name of the {ProductShortName} namespace.
+
The command returns output similar to the following example:
+
[source,terminal]
----
mta-ui-mta-proxy-7d6f9b8c4-xk2pq 1/1 Running 0 2d
----

. Stream the current log output from the proxy pod:
+
[source,terminal,subs="+quotes"]
----
$ oc logs -f __<proxy_pod_name>__ -n __<mta_namespace>__
----
+
where:
+
`<proxy_pod_name>`:: Specifies the name of the proxy pod.

. To retrieve logs from a previous (terminated) proxy pod, add the `--previous` flag to the command:
+
[source,terminal,subs="+quotes"]
----
$ oc logs --previous __<proxy_pod_name>__ -n __<mta_namespace>__
Comment on lines +50 to +54

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

OpenShift oc logs --previous previous container stopped terminated pod replaced

💡 Result:

To retrieve logs from a container that has crashed or restarted within an OpenShift pod, use the --previous (or -p) flag with the oc logs command [1][2]. The command syntax is: oc logs --previous If the pod contains multiple containers, you must specify the container name using the -c flag [3][2]: oc logs -c --previous Key considerations regarding this command: 1. Scope of the flag: The --previous flag retrieves logs from the most recent terminated instance of a container within the current pod [1][4][2]. If a container has crashed multiple times, this flag typically only provides access to the immediately preceding instance [3]. 2. Pod replacement: The --previous flag relies on the container being part of the same pod instance [3]. If an entire pod is terminated, deleted, or replaced by a higher-level controller (such as a Deployment or ReplicaSet), the logs from the previous pod are generally lost, as they are not persisted by the cluster once the pod object is removed [3][5]. OpenShift pods are treated as expendable and immutable; when configurations change or a pod fails, the system terminates the existing pod and creates a new one [5]. 3. Troubleshooting tips: If you receive an error stating that the previous container could not be found, it is likely because: - The container has not actually restarted (it is still on its first run). - The entire pod was replaced, meaning the previous log stream is no longer accessible via oc logs [3]. - You are attempting to access logs from a pod that has been garbage-collected or pruned [6]. For persistent logging, especially for pods that frequently restart or are replaced, it is recommended to implement external log aggregation (e.g., via OpenShift Logging/Elasticsearch, Fluentd, or other logging sidecars) to capture and store logs before the container or pod is terminated [3].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc"
if [ -f "$file" ]; then
  echo "== file existence and relevant lines =="
  nl -ba "$file" | sed -n '40,60p'
else
  echo "missing $file"
fi

Repository: migtools/mta-documentation

Length of output: 244

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc"
if [ -f "$file" ]; then
  echo "== file existence and relevant lines =="
  awk 'NR >= 45 && NR <= 55 { printf "%5d\t%s\n", NR, $0 }' "$file"
else
  echo "missing $file"
fi

Repository: migtools/mta-documentation

Length of output: 551

Clarify the meaning of --previous.

oc logs --previous retrieves logs from the previous container instance in the current pod after a restart; it does not access logs from a replaced or deleted pod. Reword this step accordingly, or add a separate example for locating and reading an older pod if needed.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/topics/developer-lightspeed/proc_access-llm-proxy-logs.adoc` around
lines 50 - 54, Update the log-retrieval step to state that oc logs --previous
reads logs from a prior container instance in the current pod after restart, not
from a replaced or deleted pod. Reword the existing description and command
context accordingly, adding guidance for older pods only if the documentation
covers that use case.

Source: MCP tools

----

Loading