MTA-7133-LLM-Proxy-Disconnection-Network-Isolation-Procedures

[anarnold97]

---

JIRA

• MTA-7133

---

PREVIEW

• 7. LLM proxy interaction monitoring

  • 7.1. LLM proxy request and data handling

  • 7.2. Accessing LLM proxy pod logs

  • 7.3. Planned audit logging capabilities

---

DESCRIPTION

Adds emergency LLM proxy shutdown and network isolation procedures to the Developer Lightspeed guide.

New file: proc_emergency-llm-proxy-shutdown.adoc

Centralized mode (Operator / OpenShift deployments)

• Disable the LLM proxy by setting kai_llm_proxy_enabled: false in the Tackle CR
• Covers both the OpenShift web console and oc patch CLI methods
• Warns against directly scaling the Deployment to zero — the Operator will re-create it
• Documents expected developer impact: 503/connection refused errors in the IDE, standard IDE functions unaffected
• Includes re-enablement procedure once the incident is resolved

Standalone mode (manual desktop setup)

• No central proxy to disable — documents the firewall egress block approach instead
• Reference table of common LLM provider hostnames to block (OpenAI, Azure OpenAI, IBM watsonx.ai, custom)
• Notes DNS-based filtering as an alternative (Pi-hole, Cisco Umbrella, Zscaler)

Validation

• End-to-end procedure to confirm isolation is active from the IDE
• Table of expected error responses per deployment mode
• Optional curl-based network-level check

Assembly change

docs/developer-lightspeed-guide/master.adoc — new module included after con_developer-lightspeed-logs.adoc

Summary by CodeRabbit

• Documentation
  • Added an emergency LLM proxy shutdown and recovery guide.
  • Documented shutdown procedures using the web console or CLI.
  • Added guidance for standalone deployments, shutdown validation, and restoring service.
  • Described expected developer and IDE behavior while the proxy is unavailable.

[coderabbitai]

Warning

Review limit reached

@anarnold97, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 33 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e4f374a1-b14f-4a16-b6c0-6550aab32afb

📥 Commits

Reviewing files that changed from the base of the PR and between 0e2236e and 2fb11b1.

📒 Files selected for processing (7)

• docs/topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc
• docs/topics/developer-lightspeed/assembly_reenable-llm-proxy-after-incident.adoc
• docs/topics/developer-lightspeed/proc_disable-llm-proxy-cli.adoc
• docs/topics/developer-lightspeed/proc_reenable-proxy-using-cli.adoc
• docs/topics/developer-lightspeed/proc_reenable-proxy-using-web-console.adoc
• docs/topics/developer-lightspeed/proc_validate-llm-proxy-shutdown.adoc
• docs/topics/developer-lightspeed/ref_llm-proxy-developer-impact.adoc

📝 Walkthrough

Walkthrough

Adds an emergency LLM proxy shutdown assembly to the developer Lightspeed guide, with centralized and standalone isolation guidance, CLI and web console procedures, shutdown validation, developer-impact notes, and re-enablement steps.

Changes

Emergency shutdown procedure docs

Layer / File(s)	Summary
Guide inclusion and shutdown overview docs/developer-lightspeed-guide/master.adoc, docs/topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc, docs/topics/developer-lightspeed/con_llm-proxy-shutdown-overview.adoc	The master guide includes a new assembly covering centralized and standalone LLM proxy shutdown.
Centralized proxy shutdown procedures docs/topics/developer-lightspeed/proc_disable-llm-proxy-web-console.adoc, docs/topics/developer-lightspeed/proc_disable-llm-proxy-cli.adoc, docs/topics/developer-lightspeed/ref_llm-proxy-developer-impact.adoc	Documents disabling the proxy through the web console or oc patch, verifying proxy pod termination, and describing developer workstation behavior.
Shutdown validation and proxy re-enablement docs/topics/developer-lightspeed/proc_validate-llm-proxy-shutdown.adoc, docs/topics/developer-lightspeed/proc_reenable-llm-proxy.adoc	Documents endpoint reachability checks and web console or CLI workflows for restoring the proxy and testing responses.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• migtools/mta-documentation#324: Covers the related centralized LLM proxy enablement workflow using the same Tackle custom resource setting.

Suggested reviewers: mpershina, pkylas007

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is specific and matches the PR's main change: emergency LLM proxy disconnection and network isolation procedures.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/topics/developer-lightspeed/proc_emergency-llm-proxy-shutdown.adoc`:
- Around line 175-182: Correct the provider name typo in the provider list so it
reads “Azure OpenAI Service” instead of “zure OpenAI Service.” Update the
relevant entry in the developer-lightspeed emergency LLM proxy shutdown document
where the OpenAI/Azure OpenAI providers are listed, keeping the rest of the
table content unchanged.
- Around line 93-104: The verification step in
proc_emergency-llm-proxy-shutdown.adoc is misleading because `oc get pods ... |
grep proxy` will not produce the stated “No resources found...” output; update
the expected result to reflect the actual shell behavior for this command, using
the existing “Verify that no proxy pods are running” step and its “Expected
output” block as the place to adjust the wording.
- Around line 243-264: The curl validation is targeting the public
`api.openai.com` endpoint instead of the actual isolation/proxy path, so it does
not validate centralized deployments correctly. Update the `curl` example in the
emergency shutdown docs to use the proxy or workstation-specific endpoint used
by the isolation flow, and make the expected output reflect that path for both
isolated and centralized modes. Keep the guidance aligned with the surrounding
`curl` validation section so readers verify the real traffic route rather than a
single provider host.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 23625628-72fe-4a48-8b6d-a0bc320e3992

📥 Commits

Reviewing files that changed from the base of the PR and between 50da7d6 and 6a1684d.

📒 Files selected for processing (2)

• docs/developer-lightspeed-guide/master.adoc
• docs/topics/developer-lightspeed/proc_emergency-llm-proxy-shutdown.adoc

[Pkylas007]

Really minor nits and requested a clarification but thanks for the draft!

[rromannissen]

Some changes required.

[Pkylas007]

LGTM, thank you!

[mpershina]

Left a few suggestions.

[mpershina]

Left more suggestions :D
The most important one is about turning one procedure module with subsections (sub procedures) into an assembly with the individual procedures as modules. Subsections aren't allowed in procedure modules, thus, suggesting an assembly.

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/topics/developer-lightspeed/proc_disable-llm-proxy-web-console.adoc (1)

38-43: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Verify that proxy pods are fully removed.

Checking only for a non-Running status can pass while a proxy pod is still Terminating and potentially serving connections. Wait until no proxy pod exists, or document a status check that explicitly confirms deletion.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/topics/developer-lightspeed/proc_disable-llm-proxy-web-console.adoc`
around lines 38 - 43, Update the Verification steps to confirm that no pod with
“proxy” in its name remains in the {ProductShortName} namespace, rather than
only checking that it is not Running. In the relevant workload verification
instructions, require waiting for terminating proxy pods to disappear or
explicitly confirm their deletion.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc`:
- Around line 20-22: Reorder the procedure includes in the assembly document so
proc_validate-llm-proxy-shutdown.adoc appears before
proc_reenable-llm-proxy.adoc, preserving the intended workflow of isolate,
validate, then re-enable.

In `@docs/topics/developer-lightspeed/proc_disable-llm-proxy-cli.adoc`:
- Around line 46-61: Update the Verification command to explicitly detect proxy
pods in the namespace, print “No proxy pods found” when none remain, and return
a failure status when any proxy pod is present. Replace the inconsistent “No
resources found in mta namespace.” output with the matching no-proxy message
while preserving the existing namespace placeholder.

In `@docs/topics/developer-lightspeed/proc_reenable-llm-proxy.adoc`:
- Around line 17-20: Update the navigation steps around the Tackle instance to
explicitly select the {ProductShortName} namespace/project and installed
{ProductShortName} Operator before opening Tackle, matching the sequence in
proc_disable-llm-proxy-web-console.adoc. Preserve the existing Actions > Edit
Tackle and kai_llm_proxy_enabled instructions.

In `@docs/topics/developer-lightspeed/proc_validate-llm-proxy-shutdown.adoc`:
- Around line 15-37: Revise the isolation validation procedure to separate
centralized, firewall-blocked, and DNS-blocked scenarios. In centralized mode,
have the curl command target the actual LLM proxy endpoint used by the IDE and
include --fail so HTTP 503 produces curl error 22; document connection-refused
output for firewall blocking and a name-resolution failure for DNS blocking.
Update the expected-output sections and placeholders to match each scenario.

---

Outside diff comments:
In `@docs/topics/developer-lightspeed/proc_disable-llm-proxy-web-console.adoc`:
- Around line 38-43: Update the Verification steps to confirm that no pod with
“proxy” in its name remains in the {ProductShortName} namespace, rather than
only checking that it is not Running. In the relevant workload verification
instructions, require waiting for terminating proxy pods to disappear or
explicitly confirm their deletion.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d5d7cf6-9056-4bfe-a91d-711e5c9712bb

📥 Commits

Reviewing files that changed from the base of the PR and between 6a1684d and 0e2236e.

📒 Files selected for processing (8)

• docs/developer-lightspeed-guide/master.adoc
• docs/topics/developer-lightspeed/assembly_emergency-llm-proxy-shutdown.adoc
• docs/topics/developer-lightspeed/con_llm-proxy-shutdown-overview.adoc
• docs/topics/developer-lightspeed/proc_disable-llm-proxy-cli.adoc
• docs/topics/developer-lightspeed/proc_disable-llm-proxy-web-console.adoc
• docs/topics/developer-lightspeed/proc_reenable-llm-proxy.adoc
• docs/topics/developer-lightspeed/proc_validate-llm-proxy-shutdown.adoc
• docs/topics/developer-lightspeed/ref_llm-proxy-developer-impact.adoc

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/developer-lightspeed-guide/master.adoc

[anarnold97]

@mpershina - FYI created three files:

1. The Assembly File -

// Assembly included in the following assemblies:
//
// * docs/developer-lightspeed-guide/master.adoc

:_mod-docs-content-type: ASSEMBLY
[id="reenable-llm-proxy_{context}"]
= Re-enable the LLM proxy after an incident

[role="_abstract"]
Re-enable the proxy only after your security team confirms that the incident is resolved and the environment is safe.

include::proc_reenable-proxy-using-web-console.adoc[leveloffset=+1]

include::proc_reenable-proxy-using-cli.adoc[leveloffset=+1]

2. First Procedure Module (Web Console)

// Module included in the following assemblies:
//
// * docs/developer-lightspeed-guide/assembly_reenable-llm-proxy-after-incident.adoc

:_mod-docs-content-type: PROCEDURE
[id="reenable-proxy-using-web-console_{context}"]
= Re-enable the proxy by using the {ocp-short} web console

[role="_abstract"]
You can re-enable the LLM proxy by using the {ocp-short} web console to restore functionality.

.Procedure

. Click *Operators* > *Installed Operators*.
. Click the *Tackle* instance.
. Click *Actions* > *Edit Tackle*.
. Set the `kai_llm_proxy_enabled` field to `true`:
+
[source,yaml]
----
spec:
  kai_llm_proxy_enabled: true
----
. Click *Save*.
+
Wait for the proxy pod to reach the `Running` status.

3. The Second Procedure Module (CLI)

// Module included in the following assemblies:
//
// * docs/developer-lightspeed-guide/assembly_reenable-llm-proxy-after-incident.adoc

:_mod-docs-content-type: PROCEDURE
[id="reenable-proxy-using-cli_{context}"]
= Re-enable the proxy by using the CLI

[role="_abstract"]
You can re-enable the LLM proxy by using the command-line interface (CLI) to restore functionality.

.Procedure

. Patch the Tackle custom resource (CR) to enable the LLM proxy:
+
[source,terminal,subs="+quotes"]
----
$ oc patch tackle __<tackle_instance_name>__ -n __<mta_namespace>__ \
  --type=merge -p '{"spec":{"kai_llm_proxy_enabled":true}}'
----
+
where:
+
`<tackle_instance_name>`:: Specifies the name of your Tackle instance.
`<mta_namespace>`:: Specifies the name of the {ProductShortName} namespace.

.Verification

. Verify that a proxy pod is running:
+
[source,terminal,subs="+quotes"]
----
$ oc get pods -n __<mta_namespace>__ | grep proxy
----
+
where:
+
`<mta_namespace>`:: Specifies the name of the {ProductShortName} namespace.
+
. Ask a developer to submit a test query from their integrated development environment (IDE) and confirm that a valid response is returned.

[anarnold97]

@rromannissen - just need your approval before moving this to merge

thanks

[rromannissen]

LGTM!