bug: ai-proxy crashes worker with SIGILL (signal 4) on outbound HTTPS request to LLM endpoint

[namannagar89]

Current Behavior

When ai-proxy makes the outbound HTTPS request to a live LLM endpoint, the nginx worker process crashes with SIGILL (signal 4, "illegal instruction"). The client receives an empty reply, and the gateway log shows:

[alert] 1#1: worker process NN exited on signal 4

One crash per request. The worker is respawned by the master, so there's no permanent outage, but every ai-proxy request that reaches the TLS handshake to the upstream LLM kills its worker.

The crash is not provider-specific and not in the Vertex JWT-signing/token path (we initially suspected that and disproved it — see below). It reproduces whenever ai-proxy opens a real HTTPS/TLS connection to the LLM endpoint.

Expected Behavior

ai-proxy proxies the request to the LLM over HTTPS and returns the response (or a normal error), without crashing the worker.

What we observed / how we isolated it

Test	Result
provider: vertex-ai → Vertex (HTTPS)	worker exited on signal 4, empty reply
provider: openai, override.endpoint: http://127.0.0.1:1/... (unreachable, no TLS)	no crash, clean HTTP 500
provider: openai → https://api.openai.com/... (real HTTPS, dummy key)	worker exited on signal 4, empty reply (reproduced twice)
Same request after moving the gateway to a different machine family (Intel E2 → AMD T2D)	still crashes

So the trigger is the outbound HTTPS/TLS call to a live endpoint (reproduced with both openai and vertex-ai); there's no crash when the endpoint is unreachable (no TLS handshake). It reproduces on both Intel and AMD CPUs.

Note: the gateway's normal reverse-proxy TLS to other HTTPS upstreams works fine on the same nodes; only ai-proxy's outbound HTTPS path crashes.

Steps to Reproduce

1. Run APISIX 3.17.0 (image apache/apisix:3.17.0-ubuntu).
2. Create a route using ai-proxy with any provider whose endpoint is real HTTPS, e.g.:

// PUT /apisix/admin/routes/1
{
  "uri": "/ai",
  "plugins": {
    "ai-proxy": {
      "provider": "openai",
      "auth": { "header": { "Authorization": "Bearer sk-dummy" } },
      "options": { "model": "gpt-4" }
    }
  },
  "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1": 1 } }
}

3. Send a request:

curl -sv http://<gateway>/ai -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}]}'
# => curl: (52) Empty reply from server

4. Gateway log shows [alert] ... worker process NN exited on signal 4.

(A dummy key is fine — the crash happens during/around the TLS connection, before a valid API response is needed.)

Environment

• APISIX version: 3.17.0 (apache/apisix:3.17.0-ubuntu)
• OS/Platform: GKE (Ubuntu-based image); reproduced on e2-standard-4 (Intel Xeon, no AVX-512) and t2d-standard-8 (AMD EPYC Milan, no AVX-512)
• ai-proxy plugins load correctly (note: on clusters where the prometheus plugin is disabled, ai-proxy fails to load because it requires the prometheus-cache shared dict — a separate issue; we declared that dict so ai-proxy loads, then hit this crash).

What we could not determine

We do not yet have a coredump/backtrace, so the exact faulting library/instruction is unknown. SIGILL (illegal instruction) on a stock amd64 build, in the outbound TLS/crypto path, on CPUs lacking AVX-512, suggests a native code path emitting an instruction the CPU doesn't support — but that's unconfirmed. Happy to gather a backtrace with guidance (e.g. recommended way to enable coredumps for the worker in the official image).

Question

Has anyone else hit a SIGILL from ai-proxy's outbound HTTPS path, particularly on non-AVX-512 CPUs? Any guidance on capturing a usable backtrace from the official image would help pinpoint the fix.

[namannagar89]

Correction / refinement: not HTTPS-specific — reproduces over plain HTTP too

Follow-up testing shows the crash is not tied to TLS/HTTPS or to the Vertex token path. It reproduces whenever ai-proxy successfully connects to a reachable upstream and processes the request/response, regardless of scheme:

Upstream	Result
Unreachable endpoint (http://127.0.0.1:1, connection refused)	no crash (graceful HTTP 500)
Reachable HTTP endpoint (in-cluster echo server returning OpenAI-style JSON)	worker exited on signal 4 (reproduced 2/2)
Reachable HTTPS endpoint (api.openai.com, Vertex)	worker exited on signal 4

So the trigger is a successful upstream connection + ai-proxy's request/response handling, not the TLS handshake and not Vertex JWT signing.

Simplified, provider-agnostic reproduction (no real LLM or TLS needed)

1. Run any plain-HTTP server that returns an OpenAI-style JSON body, e.g.:

   http-echo -listen=:5678 -text='{"id":"x","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}],"usage":{"total_tokens":2}}'
   

2. Route it through ai-proxy over HTTP:

   {"uri":"/ai","plugins":{"ai-proxy":{"provider":"openai","auth":{"header":{"Authorization":"Bearer x"}},
     "options":{"model":"gpt-4"},"override":{"endpoint":"http://<echo-host>:5678/v1/chat/completions"}}},
     "upstream":{"type":"roundrobin","nodes":{"127.0.0.1:1":1}}}

3. POST /ai with {"messages":[{"role":"user","content":"hi"}]} → nginx worker exited on signal 4, client gets empty reply.

This should make it reproducible without any external provider/credentials. Title/description effectively broaden from "outbound HTTPS request" to "any successful outbound request the plugin then processes."