fix(ai): fall back to default gateway when swarm Bifrost is unreachable

[Evanfeenstra]

Problem

When a workspace's swarm has an expired TLS cert (or is otherwise unreachable), the entire chat turn dies with CERT_HAS_EXPIRED:

[runCanvasAgent] streamText error: Failed after 3 attempts.
  Last error: Cannot connect to API: certificate has expired
  url: 'https://swarm38.sphinx.chat:8181/anthropic/v1/messages'

The LLM inference call routes through one gateway — the primary workspace's swarm Bifrost proxy (bifrost.baseUrl). If that swarm is down, there's no fallback, so the turn fails even when the default gateway is perfectly healthy. (The concept pre-fetch already skips dead swarms; the LLM call didn't.)

Fix

Pre-flight the swarm Bifrost gateway before committing to it:

• Reachable (any resolved HTTP response, even 404/401 → TLS+TCP handshake OK): keep the swarm route.
• Unreachable (CERT_HAS_EXPIRED, ECONNREFUSED, timeout, fetch failed): drop the entire Bifrost bundle (baseUrl + VK + macaroon) and fall back to the plain default gateway — the same path a non-Bifrost workspace already uses (default key, no override).

We probe instead of catching a mid-stream streamText error because by the time that surfaces, the 200 + SSE response is already returned to the client and the stream can't be restarted.

This is not disabling cert verification — it routes around the broken swarm. The swarm cert still needs renewing; this just stops it from taking down chat.

Changes

• src/lib/ai/provider.ts — shared isGatewayReachable(baseUrl) probe (3s timeout).
• src/lib/ai/runCanvasAgent.ts — main canvas stream falls back to the default gateway when the swarm gateway is unreachable (covers /api/ask/quick and /api/ask/sync). Adds a bifrostFellBack flag to the streamText log.
• src/services/canvas-turn-enrichments.ts — follow-up-questions generateObject (which failed the same way) gets the same fallback.

Verification

Probed the real endpoint https://swarm38.sphinx.chat:8181:

• Node fetch to root resolves HTTP 200 → isGatewayReachable() returns true → no false fallbacks on a healthy gateway; no need for a special /health path.
• When the cert is expired, Node fetch rejects → probe returns false → fall back to default gateway.

Tradeoffs

• Adds one cheap probe round-trip per Bifrost-routed turn (only for workspaces in the BIFROST_ENABLED allow-list). ~50–200ms on a healthy swarm; fails fast and rescues the turn on a dead one.
• Only falls back on connectivity/TLS failures. If the gateway connects but the upstream model 500s, we keep the swarm route and surface the real error.

[github-actions]

Test environment is now live.

View it at: https://hive-preview-2.sphinx.chat

Database expires at: Jun 28, 2026, 5:41 AM UTC