From db8c363787c4ae25fe44a952b72acf4df4ff28d5 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 09:49:39 +0200
Subject: [PATCH 01/11] Add backend client observability plan

---
 .../backend-client-observability-plan.md      | 561 ++++++++++++++++++
 1 file changed, 561 insertions(+)
 create mode 100644 .sisyphus/plans/backend-client-observability-plan.md

diff --git a/.sisyphus/plans/backend-client-observability-plan.md b/.sisyphus/plans/backend-client-observability-plan.md
new file mode 100644
index 000000000..a9c889d60
--- /dev/null
+++ b/.sisyphus/plans/backend-client-observability-plan.md
@@ -0,0 +1,561 @@
+# Backend Client Observability Implementation Plan
+
+## Context
+
+We want backend-side observability for Vortex partner/API-client issues before adding alerts and dashboards. The immediate goal is to record reliable metrics and structured logs for partner-facing API flows without changing existing quote/ramp behavior.
+
+Primary flows to observe:
+
+- Quote creation and retrieval
+- Ramp register
+- Ramp update
+- Ramp start
+- Ramp status
+- Ramp error log retrieval
+- Auth/config failures around these flows
+
+The monitoring should help answer:
+
+- Which partner/API client is having issues?
+- Which operation is failing?
+- Is the issue auth/config, validation, expired quote, missing presigned transactions, backend error, or async ramp execution?
+- Is this isolated to one partner or global?
+- Which request IDs, quote IDs, and ramp IDs can be used for debugging?
+
+## Core Design Principle
+
+Observability must run alongside existing flows and must not affect business behavior.
+
+Rules:
+
+1. Do not change controller/service response bodies.
+2. Do not change HTTP statuses.
+3. Do not introduce retries or new side effects in quote/ramp flows.
+4. Do not block API responses on metric/log persistence.
+5. Do not allow observability failures to throw into request handling.
+6. Do not log secrets or sensitive user/payment/KYC data.
+
+All observability writes must be best-effort. If persistence fails, the API request should continue exactly as it does today.
+
+## Persistence Decision
+
+### Recommendation
+
+Use an append-only database table for persistent operational events, plus existing Winston logs for detailed application logging. Do not use Supabase Storage as the primary metrics store.
+
+### Why not only in-memory metrics?
+
+In-memory counters reset on server restart and are not enough if we want historical partner health, later dashboarding, or investigation across deployments. They are still useful for Prometheus-style scraping later, but they should not be the only source of partner health history.
+
+### Supabase Storage bucket option
+
+Supabase Storage is better for raw archive files, such as daily NDJSON log exports. It is not ideal as the primary backend monitoring store because:
+
+- querying by partner, operation, status, and time window is awkward
+- alert/dashboard queries require downloading/parsing files or secondary indexing
+- concurrent append patterns are less natural than database inserts
+- retention and aggregation become custom jobs
+
+Storage can be added later as a cold archive, but not as the main operational source.
+
+### Extra table option
+
+An append-only table is better for this phase because:
+
+- it survives restarts
+- it supports per-partner dashboard queries
+- it supports alert queries over time windows
+- it is easy to join conceptually with partner/quote/ramp identifiers
+- it can be indexed by `createdAt`, `partnerId`, `operation`, `status`, and `errorType`
+
+### Important caveat
+
+Do not write one table row for every noisy low-value event forever without retention. Start with partner-facing operations only, and add retention/aggregation before traffic grows significantly.
+
+Recommended initial retention:
+
+- raw operational events: 30-90 days
+- later aggregated daily/hourly summaries: longer retention if needed
+
+## Proposed Data Model
+
+Create a new Sequelize model and migration for an append-only table, tentatively named `ApiClientEvent`.
+
+Suggested table: `api_client_events`
+
+Fields:
+
+```text
+id UUID primary key
+request_id string nullable/indexed
+operation string indexed
+status string indexed                 # success | failure
+http_status integer nullable indexed
+error_type string nullable indexed
+error_message string nullable          # sanitized, short message only
+partner_id UUID nullable indexed
+partner_name string nullable indexed
+api_key_prefix string nullable indexed # never secret key value
+user_id UUID nullable indexed
+quote_id UUID nullable indexed
+ramp_id UUID nullable indexed
+ramp_type string nullable
+network string nullable
+payment_method string nullable
+duration_ms integer nullable
+metadata jsonb nullable                # sanitized low-volume details only
+created_at timestamp indexed
+updated_at timestamp
+```
+
+Notes:
+
+- `quote_id` and `ramp_id` are acceptable in the table, but should not be metric labels.
+- `error_message` must be sanitized and short. Prefer stable `error_type` for dashboards.
+- `metadata` must never contain request bodies, raw headers, tax IDs, PIX keys, QR codes, KYC data, API secrets, or ephemeral secrets.
+- Use partner name/ID and API key prefix only for attribution.
+
+## Metrics Shape
+
+Persistent events support historical querying. In addition, expose or prepare metric-style counters/histograms through a small abstraction so a Prometheus/Grafana or Datadog integration can be added later.
+
+Metric names to prepare:
+
+```text
+vortex_api_client_requests_total{operation,partner,status,http_status,error_type}
+vortex_api_client_request_duration_seconds{operation,partner,status}
+vortex_api_auth_failures_total{route,auth_error_type,partner}
+vortex_ramp_operation_failures_total{partner,operation,error_type}
+```
+
+Metric label rules:
+
+- OK: partner slug/name/id, operation, status, HTTP status, error type, environment
+- Not OK: ramp ID, quote ID, user ID, wallet address, tax ID, PIX key, request ID
+
+High-cardinality identifiers belong in structured logs and the persistent event table, not labels.
+
+## Structured Logging Shape
+
+Use the existing backend logger. Add one structured warning/error log for failed partner-facing operations and optionally one compact info log for successful operations.
+
+Failure log shape:
+
+```ts
+logger.warn("Partner API operation failed", {
+  requestId,
+  operation,
+  partnerId,
+  partnerName,
+  apiKeyPrefix,
+  userId,
+  quoteId,
+  rampId,
+  httpStatus,
+  errorType,
+  errorMessage,
+  durationMs
+});
+```
+
+Success log shape:
+
+```ts
+logger.info("Partner API operation completed", {
+  requestId,
+  operation,
+  partnerId,
+  partnerName,
+  apiKeyPrefix,
+  userId,
+  quoteId,
+  rampId,
+  durationMs
+});
+```
+
+Logging guidance:
+
+- Always log failures.
+- Success logs can be sampled or omitted later if volume is too high.
+- Never log raw request bodies or headers.
+- Never log API secret keys, KYC data, PIX data, QR codes, tax IDs, or ephemeral secret material.
+
+## Error Classification
+
+Add a helper that maps known errors into stable `error_type` values. Dashboards and alerts should use these stable categories instead of raw error messages.
+
+Initial categories:
+
+```text
+none
+validation_error
+auth_missing_api_key
+auth_invalid_api_key
+auth_invalid_public_key
+auth_partner_mismatch
+auth_inactive_partner
+ownership_denied
+quote_not_found
+quote_expired
+quote_consumed
+invalid_ephemerals
+invalid_presigned_transactions
+missing_presigned_transactions
+time_window_exceeded
+ramp_not_found
+ramp_not_updatable
+ramp_not_in_initial_state
+service_unavailable
+provider_error
+internal_error
+unknown_error
+```
+
+Classification should be conservative. If unsure, classify as `unknown_error` or `internal_error` rather than parsing sensitive details.
+
+## Proposed Module Layout
+
+Add a small observability module under the API app:
+
+```text
+apps/api/src/api/observability/
+  requestContext.ts
+  apiClientEvents.model.ts        # or place model under apps/api/src/models/
+  apiClientEvent.service.ts
+  metrics.ts
+  operationLogger.ts
+  errorClassifier.ts
+  types.ts
+```
+
+Preferred model location should follow existing repo patterns. Since existing Sequelize models live in `apps/api/src/models`, the actual model should likely be:
+
+```text
+apps/api/src/models/apiClientEvent.model.ts
+```
+
+and exported from:
+
+```text
+apps/api/src/models/index.ts
+```
+
+Responsibilities:
+
+- `requestContext.ts`: generate/propagate request ID and timing context.
+- `apiClientEvent.service.ts`: best-effort event persistence.
+- `metrics.ts`: safe counters/histogram helpers or placeholders for future exporter integration.
+- `operationLogger.ts`: structured success/failure logs.
+- `errorClassifier.ts`: stable error category mapping.
+- `types.ts`: shared operation/status/event types.
+
+Every public observability function should catch and swallow its own failures after logging a local debug/warn message if appropriate.
+
+## Request Context Middleware
+
+Add middleware early in Express setup to:
+
+- read an incoming request/correlation ID header if present
+- generate a request ID if absent
+- attach it to `req`
+- add `X-Request-ID` to the response
+- record `startedAt` for duration calculation
+
+Candidate headers to accept:
+
+```text
+X-Request-ID
+X-Correlation-ID
+```
+
+If both exist, prefer `X-Request-ID`.
+
+The middleware must not require any new client behavior.
+
+## Instrumentation Points
+
+### Auth/config failures
+
+Instrument auth-related failure branches in:
+
+- `apps/api/src/api/middlewares/apiKeyAuth.ts`
+- `apps/api/src/api/middlewares/publicKeyAuth.ts`
+- `apps/api/src/api/middlewares/dualAuth.ts`
+- `apps/api/src/api/middlewares/ownershipAuth.ts`
+
+Record events for:
+
+- missing API key
+- invalid API key format
+- invalid API key
+- invalid public key
+- partner mismatch
+- inactive/expired key if distinguishable
+- ownership denied
+
+These events may not have quote/ramp context yet. Use route, partner if known, key prefix if safe, and request ID.
+
+### Quote controller
+
+Instrument:
+
+- `createQuote`
+- `createBestQuote`
+- `getQuote`
+
+Relevant files:
+
+- `apps/api/src/api/controllers/quote.controller.ts`
+- `apps/api/src/api/services/quote/index.ts`
+- quote finalization/persistence service where `QuoteTicket` is created
+
+Capture:
+
+- operation
+- partner ID/name if known
+- public API key or prefix if already stored as safe public key
+- quote ID on success
+- ramp type
+- network
+- payment method
+- duration
+- status/error type
+
+### Ramp controller/service
+
+Instrument:
+
+- `registerRamp`
+- `updateRamp`
+- `startRamp`
+- `getRampStatus`
+- `getRampErrors`
+
+Relevant files:
+
+- `apps/api/src/api/controllers/ramp.controller.ts`
+- `apps/api/src/api/services/ramp/ramp.service.ts`
+
+Capture:
+
+- operation
+- partner ID/name from authenticated request if available
+- partner ID/name via `RampState -> QuoteTicket` or `QuoteTicket` after records are loaded
+- quote ID
+- ramp ID
+- ramp type if available
+- network/payment method if available
+- duration
+- status/error type
+
+Do not alter existing service behavior. Add observation around existing success/catch paths only.
+
+### Async phase execution
+
+Do not mix async phase execution failures with synchronous partner API call failures in the first pass.
+
+Existing phase errors are already stored in `RampState.errorLogs`. Later, phase failures can emit their own separate events such as:
+
+```text
+operation = ramp_phase_execution
+phase = nablaSwap | performBrlaPayout | ...
+```
+
+For this initial plan, focus on partner-facing request operations.
+
+## Event Recording Pattern
+
+Use a best-effort helper instead of direct model writes in controllers.
+
+Example shape:
+
+```ts
+await recordApiClientEventSafe({
+  requestId,
+  operation: "ramp_start",
+  status: "failure",
+  httpStatus,
+  errorType,
+  partnerId,
+  partnerName,
+  quoteId,
+  rampId,
+  durationMs
+});
+```
+
+The helper must internally catch errors:
+
+```ts
+export async function recordApiClientEventSafe(event: ApiClientEventInput): Promise<void> {
+  try {
+    await ApiClientEvent.create(sanitizeEvent(event));
+  } catch (error) {
+    logger.warn("Failed to record API client event", { error: error instanceof Error ? error.message : String(error) });
+  }
+}
+```
+
+If we are concerned about adding DB write latency to the request path, make the helper fire-and-forget:
+
+```ts
+void recordApiClientEventSafe(event);
+```
+
+Preferred initial approach:
+
+- use fire-and-forget persistence for event rows
+- make observability failures impossible to surface to clients
+- keep event payload small
+
+Tradeoff: fire-and-forget can lose events if the process exits immediately. That is acceptable for non-critical observability and safer than blocking ramp flows.
+
+## Migration and Model Plan
+
+1. Add Sequelize migration for `api_client_events`.
+2. Add `ApiClientEvent` model.
+3. Export model from `apps/api/src/models/index.ts`.
+4. Add indexes:
+   - `created_at`
+   - `partner_id, created_at`
+   - `partner_name, created_at`
+   - `operation, created_at`
+   - `status, created_at`
+   - `error_type, created_at`
+   - `request_id`
+   - `quote_id`
+   - `ramp_id`
+5. Consider retention cleanup in a later migration/worker, not in the first implementation unless there is an existing cleanup pattern that makes it trivial.
+
+## Rollout Steps
+
+### Step 1: Foundation
+
+- Add request context middleware.
+- Add request ID response header.
+- Add observability types, error classifier, event service, and structured logging helper.
+- Add model/migration for `api_client_events`.
+
+Verification:
+
+- Existing API tests pass.
+- Manual request receives `X-Request-ID`.
+- No response body/status changes.
+
+### Step 2: Auth events
+
+- Instrument auth/config failure branches.
+- Record persistent events and structured logs for failures only.
+
+Verification:
+
+- Existing auth failures still return the same status/body.
+- Event rows are created for invalid/missing keys.
+- No secret key is logged or stored.
+
+### Step 3: Quote events
+
+- Instrument quote create/best/get success and failure.
+- Persist compact events.
+- Log failures.
+
+Verification:
+
+- Existing quote behavior unchanged.
+- Success/failure rows include operation, partner when known, quote ID on success, duration.
+
+### Step 4: Ramp request events
+
+- Instrument ramp register/update/start/status/errors success and failure.
+- Enrich partner attribution from `QuoteTicket`/`RampState` where available.
+
+Verification:
+
+- Existing ramp behavior unchanged.
+- Event rows include ramp/quote IDs where available.
+- Expected error categories are stable.
+
+### Step 5: Metric abstraction
+
+- Add safe metric helper functions mirroring event writes.
+- Keep implementation simple initially: either no-op counters plus persistent events, or in-process counters if a metrics endpoint already exists.
+- Do not block on full Prometheus/Grafana setup in this phase.
+
+Verification:
+
+- Metric helper calls cannot throw.
+- Label values are low-cardinality.
+
+### Step 6: Follow-up foundation for alerts/dashboard
+
+- Add simple query helpers for partner health if needed.
+- Example queries:
+  - failures by partner/operation over last 15 minutes
+  - zero success despite traffic
+  - top error types by partner over last 24 hours
+  - recent failed ramp IDs for partner
+
+This step can be done just before implementing alerts/dashboard.
+
+## Testing Plan
+
+Add focused tests where practical:
+
+- `errorClassifier` maps known errors/statuses to stable categories.
+- `sanitizeEvent` removes or rejects sensitive fields.
+- `recordApiClientEventSafe` swallows persistence errors.
+- request context middleware sets request ID and response header.
+
+Regression checks:
+
+- Existing quote tests pass.
+- Existing ramp tests pass.
+- Existing auth tests pass, if present.
+- `bun lint:fix`
+- `bun typecheck`
+- `bun build:backend`
+
+Follow repo guidance:
+
+- Always use `bun`.
+- Run `bun lint:fix` after code changes.
+- If shared package changes are made, run `bun build:shared` first. This plan should avoid shared changes unless necessary.
+
+## Privacy and Safety Checklist
+
+Before merging implementation, verify:
+
+- No `secretKey` or `X-API-Key` values are stored/logged.
+- No raw request headers are stored/logged.
+- No raw request bodies are stored/logged.
+- No tax IDs are stored/logged.
+- No PIX keys/destinations are stored/logged.
+- No QR codes/payment payloads are stored/logged.
+- No KYC data is stored/logged.
+- No ephemeral private keys or signed transaction payloads are stored/logged.
+- Metrics labels do not include request ID, quote ID, ramp ID, user ID, wallet addresses, or any free-form user data.
+- Observability write failures cannot affect API responses.
+
+## Open Questions for Implementation Time
+
+Resolve these by reading current code before editing:
+
+1. Exact migration naming/date convention in `apps/api/src/database/migrations`.
+2. Whether a `/metrics` endpoint or Prometheus client already exists.
+3. Exact Express request type augmentation pattern in this repo.
+4. Existing tests around quote/ramp controllers and whether event persistence should be mocked.
+5. Whether success events should be recorded for all operations initially, or only failures plus aggregate counters.
+
+Default answer for item 5: record both success and failure for partner-facing operations initially, because the dashboard needs denominators. If volume becomes an issue, add retention/aggregation rather than removing success visibility prematurely.
+
+## Success Criteria
+
+The implementation is complete when:
+
+- Partner-facing quote/ramp/auth operations create persistent sanitized operational events.
+- Failures create structured logs with request ID, operation, partner attribution when available, stable error type, and duration.
+- Existing API behavior is unchanged.
+- Event persistence failures are swallowed and cannot break quote/ramp flows.
+- Sensitive fields are not stored or logged.
+- The resulting table can answer per-partner failure-rate and recent-failure queries needed for future alerts/dashboard work.

From 5a8b1ec1b57e793df9adbf44015cf7c804d22b37 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 09:49:51 +0200
Subject: [PATCH 02/11] Add persistent API client event tracking

---
 .../apiClientEvent.service.test.ts            |  53 +++++++++
 .../observability/apiClientEvent.service.ts   | 103 ++++++++++++++++++
 .../api/observability/errorClassifier.test.ts |  27 +++++
 .../src/api/observability/errorClassifier.ts  |  52 +++++++++
 apps/api/src/api/observability/metrics.ts     |   6 +
 .../src/api/observability/operationLogger.ts  |  30 +++++
 .../api/observability/requestContext.test.ts  |  40 +++++++
 .../src/api/observability/requestContext.ts   |  37 +++++++
 apps/api/src/api/observability/types.ts       |  59 ++++++++++
 apps/api/src/config/express.ts                |   6 +-
 .../032-create-api-client-events-table.ts     |  56 ++++++++++
 apps/api/src/models/apiClientEvent.model.ts   |  99 +++++++++++++++++
 apps/api/src/models/index.ts                  |   2 +
 13 files changed, 569 insertions(+), 1 deletion(-)
 create mode 100644 apps/api/src/api/observability/apiClientEvent.service.test.ts
 create mode 100644 apps/api/src/api/observability/apiClientEvent.service.ts
 create mode 100644 apps/api/src/api/observability/errorClassifier.test.ts
 create mode 100644 apps/api/src/api/observability/errorClassifier.ts
 create mode 100644 apps/api/src/api/observability/metrics.ts
 create mode 100644 apps/api/src/api/observability/operationLogger.ts
 create mode 100644 apps/api/src/api/observability/requestContext.test.ts
 create mode 100644 apps/api/src/api/observability/requestContext.ts
 create mode 100644 apps/api/src/api/observability/types.ts
 create mode 100644 apps/api/src/database/migrations/032-create-api-client-events-table.ts
 create mode 100644 apps/api/src/models/apiClientEvent.model.ts

diff --git a/apps/api/src/api/observability/apiClientEvent.service.test.ts b/apps/api/src/api/observability/apiClientEvent.service.test.ts
new file mode 100644
index 000000000..dc0ced211
--- /dev/null
+++ b/apps/api/src/api/observability/apiClientEvent.service.test.ts
@@ -0,0 +1,53 @@
+import { afterEach, describe, expect, it, mock } from "bun:test";
+import ApiClientEvent from "../../models/apiClientEvent.model";
+import { recordApiClientEventSafe, sanitizeApiClientEvent } from "./apiClientEvent.service";
+
+describe("sanitizeApiClientEvent", () => {
+  it("removes sensitive metadata and replaces raw error messages", () => {
+    const sanitized = sanitizeApiClientEvent({
+      apiKeyPrefix: "pk_live_1234567890",
+      errorMessage: "Invalid EVM address format: 0x1234567890abcdef with taxId 12345678900",
+      errorType: "invalid_ephemerals",
+      metadata: {
+        apiKey: "pk_live_secret",
+        endpoint: "/v1/ramp/start",
+        nested: { unsafe: true },
+        taxId: "12345678900"
+      },
+      operation: "ramp_start",
+      partnerName: "p".repeat(150),
+      status: "failure"
+    });
+
+    expect(sanitized.apiKeyPrefix).toHaveLength(16);
+    expect(sanitized.errorMessage).toBe("Ephemeral account validation failed.");
+    expect(sanitized.errorMessage).not.toContain("0x1234567890abcdef");
+    expect(sanitized.errorMessage).not.toContain("12345678900");
+    expect(sanitized.errorType).toBe("invalid_ephemerals");
+    expect(sanitized.metadata).toEqual({ endpoint: "/v1/ramp/start" });
+    expect(sanitized.partnerName).toHaveLength(100);
+  });
+
+  it("defaults successful events to the none error type", () => {
+    const sanitized = sanitizeApiClientEvent({ operation: "quote_get", status: "success" });
+
+    expect(sanitized.errorType).toBe("none");
+    expect(sanitized.errorMessage).toBeNull();
+  });
+});
+
+describe("recordApiClientEventSafe", () => {
+  const originalCreate = ApiClientEvent.create;
+
+  afterEach(() => {
+    ApiClientEvent.create = originalCreate;
+  });
+
+  it("swallows persistence failures", async () => {
+    ApiClientEvent.create = mock(async () => {
+      throw new Error("database unavailable");
+    }) as typeof ApiClientEvent.create;
+
+    await expect(recordApiClientEventSafe({ operation: "quote_create", status: "failure" })).resolves.toBeUndefined();
+  });
+});
diff --git a/apps/api/src/api/observability/apiClientEvent.service.ts b/apps/api/src/api/observability/apiClientEvent.service.ts
new file mode 100644
index 000000000..91910b693
--- /dev/null
+++ b/apps/api/src/api/observability/apiClientEvent.service.ts
@@ -0,0 +1,103 @@
+import logger from "../../config/logger";
+import ApiClientEvent from "../../models/apiClientEvent.model";
+import { recordApiClientMetricsSafe } from "./metrics";
+import { logApiClientOperationSafe } from "./operationLogger";
+import { ApiClientErrorType, ApiClientEventInput } from "./types";
+
+const SENSITIVE_METADATA_KEYS = new Set([
+  "additionalData",
+  "apiKey",
+  "authorization",
+  "depositQrCode",
+  "ephemeralAccounts",
+  "ibanPaymentData",
+  "pixDestination",
+  "presignedTxs",
+  "rawBody",
+  "receiverTaxId",
+  "secretKey",
+  "signingAccounts",
+  "taxId",
+  "walletAddress",
+  "x-api-key"
+]);
+
+export function observeApiClientEvent(event: ApiClientEventInput): void {
+  try {
+    const sanitizedEvent = sanitizeApiClientEvent(event);
+    logApiClientOperationSafe(sanitizedEvent);
+    recordApiClientMetricsSafe(sanitizedEvent);
+    void recordApiClientEventSafe(sanitizedEvent);
+  } catch (error) {
+    logger.warn("Failed to observe API client event", { error: error instanceof Error ? error.message : String(error) });
+  }
+}
+
+export async function recordApiClientEventSafe(event: ApiClientEventInput): Promise<void> {
+  try {
+    await ApiClientEvent.create(sanitizeApiClientEvent(event));
+  } catch (error) {
+    logger.warn("Failed to record API client event", { error: error instanceof Error ? error.message : String(error) });
+  }
+}
+
+export function sanitizeApiClientEvent(event: ApiClientEventInput): ApiClientEventInput {
+  const errorType = event.errorType || (event.status === "success" ? "none" : "unknown_error");
+  return {
+    ...event,
+    apiKeyPrefix: trimString(event.apiKeyPrefix, 16),
+    errorMessage: getSafeErrorMessage(errorType),
+    errorType,
+    metadata: sanitizeMetadata(event.metadata),
+    partnerName: trimString(event.partnerName, 100),
+    status: event.status
+  };
+}
+
+function sanitizeMetadata(metadata: Record<string, unknown> | null | undefined): Record<string, unknown> | null {
+  if (!metadata) return null;
+
+  const sanitized: Record<string, unknown> = {};
+  for (const [key, value] of Object.entries(metadata)) {
+    if (SENSITIVE_METADATA_KEYS.has(key) || typeof value === "object") continue;
+    sanitized[key] = typeof value === "string" ? trimString(value, 100) : value;
+  }
+
+  return Object.keys(sanitized).length > 0 ? sanitized : null;
+}
+
+function trimString(value: string | null | undefined, maxLength: number): string | null {
+  if (!value) return null;
+  return value.slice(0, maxLength);
+}
+
+function getSafeErrorMessage(errorType: ApiClientErrorType): string | null {
+  if (errorType === "none") return null;
+
+  const messages: Record<ApiClientErrorType, string | null> = {
+    auth_inactive_partner: "Partner authentication is inactive.",
+    auth_invalid_api_key: "API authentication failed.",
+    auth_invalid_public_key: "Public API key validation failed.",
+    auth_missing_api_key: "API authentication is missing.",
+    auth_partner_mismatch: "Authenticated partner does not match the requested partner.",
+    internal_error: "Internal server error.",
+    invalid_ephemerals: "Ephemeral account validation failed.",
+    invalid_presigned_transactions: "Presigned transaction validation failed.",
+    missing_presigned_transactions: "Required presigned transactions are missing.",
+    none: null,
+    ownership_denied: "Authenticated principal does not own the resource.",
+    provider_error: "Provider operation failed.",
+    quote_consumed: "Quote is already consumed.",
+    quote_expired: "Quote is expired.",
+    quote_not_found: "Quote was not found.",
+    ramp_not_found: "Ramp was not found.",
+    ramp_not_in_initial_state: "Ramp is not in an updatable state.",
+    ramp_not_updatable: "Ramp cannot be updated.",
+    service_unavailable: "Service is unavailable.",
+    time_window_exceeded: "Allowed time window was exceeded.",
+    unknown_error: "Unknown API client operation error.",
+    validation_error: "Request validation failed."
+  };
+
+  return messages[errorType];
+}
diff --git a/apps/api/src/api/observability/errorClassifier.test.ts b/apps/api/src/api/observability/errorClassifier.test.ts
new file mode 100644
index 000000000..eb755baff
--- /dev/null
+++ b/apps/api/src/api/observability/errorClassifier.test.ts
@@ -0,0 +1,27 @@
+import { describe, expect, it } from "bun:test";
+import httpStatus from "http-status";
+import { APIError } from "../errors/api-error";
+import { classifyApiClientError } from "./errorClassifier";
+
+describe("classifyApiClientError", () => {
+  it("classifies common quote and ramp errors", () => {
+    expect(classifyApiClientError(new APIError({ message: "Quote not found", status: httpStatus.NOT_FOUND }))).toBe(
+      "quote_not_found"
+    );
+    expect(classifyApiClientError(new APIError({ message: "Quote has expired", status: httpStatus.BAD_REQUEST }))).toBe(
+      "quote_expired"
+    );
+    expect(classifyApiClientError(new APIError({ message: "No presigned transactions found", status: httpStatus.BAD_REQUEST }))).toBe(
+      "missing_presigned_transactions"
+    );
+  });
+
+  it("classifies auth and ownership failures", () => {
+    expect(classifyApiClientError(new APIError({ message: "Authentication required", status: httpStatus.UNAUTHORIZED }))).toBe(
+      "auth_missing_api_key"
+    );
+    expect(classifyApiClientError(new APIError({ message: "Authenticated user does not own this ramp", status: httpStatus.FORBIDDEN }))).toBe(
+      "ownership_denied"
+    );
+  });
+});
diff --git a/apps/api/src/api/observability/errorClassifier.ts b/apps/api/src/api/observability/errorClassifier.ts
new file mode 100644
index 000000000..6ebbb1c33
--- /dev/null
+++ b/apps/api/src/api/observability/errorClassifier.ts
@@ -0,0 +1,52 @@
+import httpStatus from "http-status";
+import { APIError } from "../errors/api-error";
+import { ApiClientErrorType } from "./types";
+
+export function classifyApiClientError(error: unknown, fallbackStatus?: number | null): ApiClientErrorType {
+  const message = getErrorMessage(error).toLowerCase();
+  const status = error instanceof APIError ? error.status : fallbackStatus;
+
+  if (message.includes("authentication required")) return "auth_missing_api_key";
+  if (message.includes("does not own") || message.includes("ownership")) return "ownership_denied";
+  if (status === httpStatus.FORBIDDEN) return "ownership_denied";
+  if (status === httpStatus.UNAUTHORIZED) return "auth_invalid_api_key";
+  if (status === httpStatus.SERVICE_UNAVAILABLE) return "service_unavailable";
+
+  if (status === httpStatus.BAD_REQUEST) {
+    if (message.includes("expired")) return "quote_expired";
+    if (message.includes("no presigned transactions")) return "missing_presigned_transactions";
+    if (message.includes("presigned")) return "invalid_presigned_transactions";
+    if (message.includes("ephemeral")) return "invalid_ephemerals";
+    if (message.includes("time window")) return "time_window_exceeded";
+    return "validation_error";
+  }
+
+  if (status === httpStatus.NOT_FOUND) {
+    if (message.includes("quote")) return "quote_not_found";
+    if (message.includes("ramp")) return "ramp_not_found";
+  }
+
+  if (status === httpStatus.CONFLICT) {
+    if (message.includes("quote")) return "quote_consumed";
+    if (message.includes("initial") || message.includes("allows updates")) return "ramp_not_in_initial_state";
+    return "ramp_not_updatable";
+  }
+
+  if (status && status >= 500) return "internal_error";
+
+  if (message.includes("quote not found")) return "quote_not_found";
+  if (message.includes("ramp not found")) return "ramp_not_found";
+  if (message.includes("quote has expired") || message.includes("quote is expired")) return "quote_expired";
+  if (message.includes("quote already consumed") || message.includes("quote is consumed")) return "quote_consumed";
+  if (message.includes("no presigned transactions")) return "missing_presigned_transactions";
+  if (message.includes("presigned")) return "invalid_presigned_transactions";
+  if (message.includes("ephemeral")) return "invalid_ephemerals";
+  if (message.includes("time window")) return "time_window_exceeded";
+  if (message.includes("provider") || message.includes("failed to calculate quote")) return "provider_error";
+
+  return "unknown_error";
+}
+
+export function getErrorMessage(error: unknown): string {
+  return error instanceof Error ? error.message : String(error);
+}
diff --git a/apps/api/src/api/observability/metrics.ts b/apps/api/src/api/observability/metrics.ts
new file mode 100644
index 000000000..cb82505bc
--- /dev/null
+++ b/apps/api/src/api/observability/metrics.ts
@@ -0,0 +1,6 @@
+import { ApiClientEventInput } from "./types";
+
+export function recordApiClientMetricsSafe(_event: ApiClientEventInput): void {
+  // Persistent api_client_events rows are the durable metric source for this phase.
+  // This hook is intentionally kept as a safe extension point for Prometheus/Datadog exporters.
+}
diff --git a/apps/api/src/api/observability/operationLogger.ts b/apps/api/src/api/observability/operationLogger.ts
new file mode 100644
index 000000000..1fa39b0f0
--- /dev/null
+++ b/apps/api/src/api/observability/operationLogger.ts
@@ -0,0 +1,30 @@
+import logger from "../../config/logger";
+import { ApiClientEventInput } from "./types";
+
+export function logApiClientOperationSafe(event: ApiClientEventInput): void {
+  try {
+    const payload = {
+      apiKeyPrefix: event.apiKeyPrefix,
+      durationMs: event.durationMs,
+      errorMessage: event.errorMessage,
+      errorType: event.errorType,
+      httpStatus: event.httpStatus,
+      operation: event.operation,
+      partnerId: event.partnerId,
+      partnerName: event.partnerName,
+      quoteId: event.quoteId,
+      rampId: event.rampId,
+      requestId: event.requestId,
+      userId: event.userId
+    };
+
+    if (event.status === "failure") {
+      logger.warn("Partner API operation failed", payload);
+      return;
+    }
+
+    logger.info("Partner API operation completed", payload);
+  } catch (error) {
+    logger.warn("Failed to log API client operation", { error: error instanceof Error ? error.message : String(error) });
+  }
+}
diff --git a/apps/api/src/api/observability/requestContext.test.ts b/apps/api/src/api/observability/requestContext.test.ts
new file mode 100644
index 000000000..a2bdb9713
--- /dev/null
+++ b/apps/api/src/api/observability/requestContext.test.ts
@@ -0,0 +1,40 @@
+import { describe, expect, it } from "bun:test";
+import { NextFunction, Request, Response } from "express";
+import { getRequestDurationMs, requestContext } from "./requestContext";
+
+describe("requestContext", () => {
+  it("uses an incoming request ID and returns it in the response header", () => {
+    const req = { headers: { "x-request-id": "req-123" } } as unknown as Request;
+    const headers: Record<string, string> = {};
+    const res = { setHeader: (key: string, value: string) => { headers[key] = value; } } as Response;
+    let calledNext = false;
+    const next: NextFunction = () => { calledNext = true; };
+
+    requestContext(req, res, next);
+
+    expect(req.requestId).toBe("req-123");
+    expect(headers["X-Request-ID"]).toBe("req-123");
+    expect(req.requestStartedAt).toBeNumber();
+    expect(calledNext).toBe(true);
+  });
+
+  it.each(["x".repeat(129), "request id with spaces"])("rejects unsafe incoming request ID %p", requestId => {
+    const req = { headers: { "x-request-id": requestId } } as unknown as Request;
+    const headers: Record<string, string> = {};
+    const res = { setHeader: (key: string, value: string) => { headers[key] = value; } } as Response;
+
+    requestContext(req, res, () => {});
+
+    expect(req.requestId).toBeDefined();
+    expect(req.requestId).not.toBe(requestId);
+    expect(req.requestId || "").toHaveLength(36);
+    expect(headers["X-Request-ID"]).toBe(req.requestId || "");
+  });
+
+  it("calculates request duration when a start time exists", () => {
+    const duration = getRequestDurationMs({ requestStartedAt: Date.now() - 5 });
+
+    expect(duration).not.toBeNull();
+    expect(duration).toBeGreaterThanOrEqual(0);
+  });
+});
diff --git a/apps/api/src/api/observability/requestContext.ts b/apps/api/src/api/observability/requestContext.ts
new file mode 100644
index 000000000..322a6cbf5
--- /dev/null
+++ b/apps/api/src/api/observability/requestContext.ts
@@ -0,0 +1,37 @@
+import { randomUUID } from "node:crypto";
+import { NextFunction, Request, Response } from "express";
+
+const MAX_REQUEST_ID_LENGTH = 128;
+const SAFE_REQUEST_ID_PATTERN = /^[A-Za-z0-9._:-]+$/;
+
+declare global {
+  // biome-ignore lint/style/noNamespace: Express request augmentation follows the existing backend pattern.
+  namespace Express {
+    interface Request {
+      requestId?: string;
+      requestStartedAt?: number;
+    }
+  }
+}
+
+export function requestContext(req: Request, res: Response, next: NextFunction): void {
+  const requestId =
+    getSafeRequestId(req.headers["x-request-id"]) || getSafeRequestId(req.headers["x-correlation-id"]) || randomUUID();
+
+  req.requestId = requestId;
+  req.requestStartedAt = Date.now();
+  res.setHeader("X-Request-ID", requestId);
+
+  next();
+}
+
+export function getRequestDurationMs(req: { requestStartedAt?: number }): number | null {
+  if (!req.requestStartedAt) return null;
+  return Date.now() - req.requestStartedAt;
+}
+
+function getSafeRequestId(value: string | string[] | undefined): string | null {
+  const requestId = Array.isArray(value) ? value[0] : value;
+  if (!requestId || requestId.length > MAX_REQUEST_ID_LENGTH || !SAFE_REQUEST_ID_PATTERN.test(requestId)) return null;
+  return requestId;
+}
diff --git a/apps/api/src/api/observability/types.ts b/apps/api/src/api/observability/types.ts
new file mode 100644
index 000000000..b4f9f4fb3
--- /dev/null
+++ b/apps/api/src/api/observability/types.ts
@@ -0,0 +1,59 @@
+export type ApiClientOperation =
+  | "auth_api_key"
+  | "auth_public_key"
+  | "auth_dual"
+  | "auth_ownership"
+  | "quote_create"
+  | "quote_create_best"
+  | "quote_get"
+  | "ramp_register"
+  | "ramp_update"
+  | "ramp_start"
+  | "ramp_status"
+  | "ramp_errors";
+
+export type ApiClientEventStatus = "success" | "failure";
+
+export type ApiClientErrorType =
+  | "none"
+  | "validation_error"
+  | "auth_missing_api_key"
+  | "auth_invalid_api_key"
+  | "auth_invalid_public_key"
+  | "auth_partner_mismatch"
+  | "auth_inactive_partner"
+  | "ownership_denied"
+  | "quote_not_found"
+  | "quote_expired"
+  | "quote_consumed"
+  | "invalid_ephemerals"
+  | "invalid_presigned_transactions"
+  | "missing_presigned_transactions"
+  | "time_window_exceeded"
+  | "ramp_not_found"
+  | "ramp_not_updatable"
+  | "ramp_not_in_initial_state"
+  | "service_unavailable"
+  | "provider_error"
+  | "internal_error"
+  | "unknown_error";
+
+export interface ApiClientEventInput {
+  requestId?: string | null;
+  operation: ApiClientOperation;
+  status: ApiClientEventStatus;
+  httpStatus?: number | null;
+  errorType?: ApiClientErrorType | null;
+  errorMessage?: string | null;
+  partnerId?: string | null;
+  partnerName?: string | null;
+  apiKeyPrefix?: string | null;
+  userId?: string | null;
+  quoteId?: string | null;
+  rampId?: string | null;
+  rampType?: string | null;
+  network?: string | null;
+  paymentMethod?: string | null;
+  durationMs?: number | null;
+  metadata?: Record<string, unknown> | null;
+}
diff --git a/apps/api/src/config/express.ts b/apps/api/src/config/express.ts
index 7fbb6d031..5fc82df5c 100644
--- a/apps/api/src/config/express.ts
+++ b/apps/api/src/config/express.ts
@@ -9,6 +9,7 @@ import methodOverride from "method-override";
 import morgan from "morgan";
 
 import { converter, handler, notFound } from "../api/middlewares/error";
+import { requestContext } from "../api/observability/requestContext";
 import routes from "../api/routes/v1";
 
 import { config } from "./vars";
@@ -25,7 +26,7 @@ const app = express();
 // enable CORS - Cross Origin Resource Sharing
 app.use(
   cors({
-    allowedHeaders: ["Content-Type", "Authorization"],
+    allowedHeaders: ["Content-Type", "Authorization", "X-API-Key", "X-Request-ID", "X-Correlation-ID"],
     credentials: true,
     maxAge: 86400, // Cache preflight requests for 24 hours
     methods: "GET,HEAD,PUT,PATCH,POST,DELETE", // Explicitly list allowed headers
@@ -55,6 +56,9 @@ app.use(limiter);
 // parse cookies
 app.use(cookieParser());
 
+// attach request IDs before request logging and route handling
+app.use(requestContext);
+
 // request logging. dev: console | production: file
 app.use(morgan(logs));
 
diff --git a/apps/api/src/database/migrations/032-create-api-client-events-table.ts b/apps/api/src/database/migrations/032-create-api-client-events-table.ts
new file mode 100644
index 000000000..cd4514bb7
--- /dev/null
+++ b/apps/api/src/database/migrations/032-create-api-client-events-table.ts
@@ -0,0 +1,56 @@
+import { DataTypes, QueryInterface } from "sequelize";
+
+export async function up(queryInterface: QueryInterface): Promise<void> {
+  await queryInterface.createTable("api_client_events", {
+    api_key_prefix: { allowNull: true, type: DataTypes.STRING(16) },
+    created_at: { allowNull: false, defaultValue: DataTypes.NOW, type: DataTypes.DATE },
+    duration_ms: { allowNull: true, type: DataTypes.INTEGER },
+    error_message: { allowNull: true, type: DataTypes.STRING(300) },
+    error_type: { allowNull: true, type: DataTypes.STRING(64) },
+    http_status: { allowNull: true, type: DataTypes.INTEGER },
+    id: { defaultValue: DataTypes.UUIDV4, primaryKey: true, type: DataTypes.UUID },
+    metadata: { allowNull: true, type: DataTypes.JSONB },
+    network: { allowNull: true, type: DataTypes.STRING(32) },
+    operation: { allowNull: false, type: DataTypes.STRING(64) },
+    partner_id: { allowNull: true, type: DataTypes.UUID },
+    partner_name: { allowNull: true, type: DataTypes.STRING(100) },
+    payment_method: { allowNull: true, type: DataTypes.STRING(32) },
+    quote_id: { allowNull: true, type: DataTypes.UUID },
+    ramp_id: { allowNull: true, type: DataTypes.UUID },
+    ramp_type: { allowNull: true, type: DataTypes.STRING(16) },
+    request_id: { allowNull: true, type: DataTypes.STRING(128) },
+    status: { allowNull: false, type: DataTypes.STRING(16) },
+    updated_at: { allowNull: false, defaultValue: DataTypes.NOW, type: DataTypes.DATE },
+    user_id: { allowNull: true, type: DataTypes.UUID }
+  });
+
+  await queryInterface.addIndex("api_client_events", {
+    fields: [{ name: "created_at", order: "DESC" }],
+    name: "idx_api_client_events_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["partner_id", "created_at"], {
+    name: "idx_api_client_events_partner_id_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["partner_name", "created_at"], {
+    name: "idx_api_client_events_partner_name_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["operation", "created_at"], {
+    name: "idx_api_client_events_operation_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["status", "created_at"], {
+    name: "idx_api_client_events_status_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["error_type", "created_at"], {
+    name: "idx_api_client_events_error_type_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["api_key_prefix", "created_at"], {
+    name: "idx_api_client_events_api_key_prefix_created_at"
+  });
+  await queryInterface.addIndex("api_client_events", ["request_id"], { name: "idx_api_client_events_request_id" });
+  await queryInterface.addIndex("api_client_events", ["quote_id"], { name: "idx_api_client_events_quote_id" });
+  await queryInterface.addIndex("api_client_events", ["ramp_id"], { name: "idx_api_client_events_ramp_id" });
+}
+
+export async function down(queryInterface: QueryInterface): Promise<void> {
+  await queryInterface.dropTable("api_client_events");
+}
diff --git a/apps/api/src/models/apiClientEvent.model.ts b/apps/api/src/models/apiClientEvent.model.ts
new file mode 100644
index 000000000..c989fb25d
--- /dev/null
+++ b/apps/api/src/models/apiClientEvent.model.ts
@@ -0,0 +1,99 @@
+import { DataTypes, Model, Optional } from "sequelize";
+import { ApiClientErrorType, ApiClientEventStatus, ApiClientOperation } from "../api/observability/types";
+import sequelize from "../config/database";
+
+export interface ApiClientEventAttributes {
+  id: string;
+  requestId: string | null;
+  operation: ApiClientOperation;
+  status: ApiClientEventStatus;
+  httpStatus: number | null;
+  errorType: ApiClientErrorType | null;
+  errorMessage: string | null;
+  partnerId: string | null;
+  partnerName: string | null;
+  apiKeyPrefix: string | null;
+  userId: string | null;
+  quoteId: string | null;
+  rampId: string | null;
+  rampType: string | null;
+  network: string | null;
+  paymentMethod: string | null;
+  durationMs: number | null;
+  metadata: Record<string, unknown> | null;
+  createdAt: Date;
+  updatedAt: Date;
+}
+
+export type ApiClientEventCreationAttributes = Optional<ApiClientEventAttributes, "id" | "createdAt" | "updatedAt">;
+
+class ApiClientEvent
+  extends Model<ApiClientEventAttributes, ApiClientEventCreationAttributes>
+  implements ApiClientEventAttributes
+{
+  declare id: string;
+  declare requestId: string | null;
+  declare operation: ApiClientOperation;
+  declare status: ApiClientEventStatus;
+  declare httpStatus: number | null;
+  declare errorType: ApiClientErrorType | null;
+  declare errorMessage: string | null;
+  declare partnerId: string | null;
+  declare partnerName: string | null;
+  declare apiKeyPrefix: string | null;
+  declare userId: string | null;
+  declare quoteId: string | null;
+  declare rampId: string | null;
+  declare rampType: string | null;
+  declare network: string | null;
+  declare paymentMethod: string | null;
+  declare durationMs: number | null;
+  declare metadata: Record<string, unknown> | null;
+  declare createdAt: Date;
+  declare updatedAt: Date;
+}
+
+ApiClientEvent.init(
+  {
+    apiKeyPrefix: { allowNull: true, field: "api_key_prefix", type: DataTypes.STRING(16) },
+    createdAt: { allowNull: false, defaultValue: DataTypes.NOW, field: "created_at", type: DataTypes.DATE },
+    durationMs: { allowNull: true, field: "duration_ms", type: DataTypes.INTEGER },
+    errorMessage: { allowNull: true, field: "error_message", type: DataTypes.STRING(300) },
+    errorType: { allowNull: true, field: "error_type", type: DataTypes.STRING(64) },
+    httpStatus: { allowNull: true, field: "http_status", type: DataTypes.INTEGER },
+    id: { defaultValue: DataTypes.UUIDV4, primaryKey: true, type: DataTypes.UUID },
+    metadata: { allowNull: true, type: DataTypes.JSONB },
+    network: { allowNull: true, type: DataTypes.STRING(32) },
+    operation: { allowNull: false, type: DataTypes.STRING(64) },
+    partnerId: { allowNull: true, field: "partner_id", type: DataTypes.UUID },
+    partnerName: { allowNull: true, field: "partner_name", type: DataTypes.STRING(100) },
+    paymentMethod: { allowNull: true, field: "payment_method", type: DataTypes.STRING(32) },
+    quoteId: { allowNull: true, field: "quote_id", type: DataTypes.UUID },
+    rampId: { allowNull: true, field: "ramp_id", type: DataTypes.UUID },
+    rampType: { allowNull: true, field: "ramp_type", type: DataTypes.STRING(16) },
+    requestId: { allowNull: true, field: "request_id", type: DataTypes.STRING(128) },
+    status: { allowNull: false, type: DataTypes.STRING(16) },
+    updatedAt: { allowNull: false, defaultValue: DataTypes.NOW, field: "updated_at", type: DataTypes.DATE },
+    userId: { allowNull: true, field: "user_id", type: DataTypes.UUID }
+  },
+  {
+    indexes: [
+      { fields: [{ name: "created_at", order: "DESC" }], name: "idx_api_client_events_created_at" },
+      { fields: ["partner_id", "created_at"], name: "idx_api_client_events_partner_id_created_at" },
+      { fields: ["partner_name", "created_at"], name: "idx_api_client_events_partner_name_created_at" },
+      { fields: ["operation", "created_at"], name: "idx_api_client_events_operation_created_at" },
+      { fields: ["status", "created_at"], name: "idx_api_client_events_status_created_at" },
+      { fields: ["error_type", "created_at"], name: "idx_api_client_events_error_type_created_at" },
+      { fields: ["api_key_prefix", "created_at"], name: "idx_api_client_events_api_key_prefix_created_at" },
+      { fields: ["request_id"], name: "idx_api_client_events_request_id" },
+      { fields: ["quote_id"], name: "idx_api_client_events_quote_id" },
+      { fields: ["ramp_id"], name: "idx_api_client_events_ramp_id" }
+    ],
+    modelName: "ApiClientEvent",
+    sequelize,
+    tableName: "api_client_events",
+    timestamps: true
+  }
+);
+
+export default ApiClientEvent;
diff --git a/apps/api/src/models/index.ts b/apps/api/src/models/index.ts
index 73ea4998d..5658dc230 100644
--- a/apps/api/src/models/index.ts
+++ b/apps/api/src/models/index.ts
@@ -1,6 +1,7 @@
 import sequelize from "../config/database";
 import AlfredPayCustomer from "./alfredPayCustomer.model";
 import Anchor from "./anchor.model";
+import ApiClientEvent from "./apiClientEvent.model";
 import ApiKey from "./apiKey.model";
 import KycLevel2 from "./kycLevel2.model";
 import MaintenanceSchedule from "./maintenanceSchedule.model";
@@ -44,6 +45,7 @@ MykoboCustomer.belongsTo(User, { as: "user", foreignKey: "userId" });
 const models = {
   AlfredPayCustomer,
   Anchor,
+  ApiClientEvent,
   ApiKey,
   KycLevel2,
   MaintenanceSchedule,

From cc6c5e76b71c55fb18cde9c408a338ff64da5702 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 09:50:05 +0200
Subject: [PATCH 03/11] Record API client observability events

---
 .../src/api/controllers/quote.controller.ts   | 114 +++++++++++++++++-
 .../src/api/controllers/ramp.controller.ts    | 113 ++++++++++++++++-
 apps/api/src/api/middlewares/apiKeyAuth.ts    |  43 +++++++
 apps/api/src/api/middlewares/dualAuth.test.ts |   1 +
 apps/api/src/api/middlewares/dualAuth.ts      |  31 +++++
 apps/api/src/api/middlewares/ownershipAuth.ts |  51 ++++++--
 apps/api/src/api/middlewares/publicKeyAuth.ts |  25 ++++
 apps/api/src/api/middlewares/validators.ts    |  28 +++++
 8 files changed, 389 insertions(+), 17 deletions(-)

diff --git a/apps/api/src/api/controllers/quote.controller.ts b/apps/api/src/api/controllers/quote.controller.ts
index d3b98cd56..f14272bdb 100644
--- a/apps/api/src/api/controllers/quote.controller.ts
+++ b/apps/api/src/api/controllers/quote.controller.ts
@@ -11,6 +11,9 @@ import { NextFunction, Request, Response } from "express";
 import httpStatus from "http-status";
 import logger from "../../config/logger";
 import { APIError } from "../errors/api-error";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { classifyApiClientError, getErrorMessage } from "../observability/errorClassifier";
+import { getRequestDurationMs } from "../observability/requestContext";
 import quoteService from "../services/quote";
 
 /**
@@ -53,9 +56,33 @@ export const createQuote = async (
       userId: req.userId
     });
 
+    observeApiClientEvent({
+      apiKeyPrefix: getSafePublicKeyPrefix(publicApiKey),
+      durationMs: getRequestDurationMs(req),
+      httpStatus: httpStatus.CREATED,
+      network,
+      operation: "quote_create",
+      partnerId: req.authenticatedPartner?.id || partnerId || null,
+      partnerName: req.authenticatedPartner?.name || publicKeyPartnerName || null,
+      paymentMethod: quote.paymentMethod,
+      quoteId: quote.id,
+      rampType,
+      requestId: req.requestId,
+      status: "success",
+      userId: req.userId || null
+    });
+
     res.status(httpStatus.CREATED).json(quote);
   } catch (error) {
-    logger.error(`Error creating quote: ${error instanceof Error ? error.message : String(error)}`);
+    logger.error("Error creating quote", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeQuoteFailure(req, "quote_create", error, {
+      apiKeyPrefix: getSafePublicKeyPrefix(req.body?.apiKey || req.validatedPublicKey?.apiKey),
+      network: getNetworkFromDestination(req.body?.rampType === RampDirection.BUY ? req.body?.to : req.body?.from),
+      partnerId: req.authenticatedPartner?.id || req.body?.partnerId || null,
+      partnerName: req.authenticatedPartner?.name || req.validatedPublicKey?.partnerName || null,
+      paymentMethod: req.body?.paymentMethod,
+      rampType: req.body?.rampType
+    });
     next(error);
   }
 };
@@ -93,9 +120,31 @@ export const createBestQuote = async (
       userId: req.userId
     });
 
+    observeApiClientEvent({
+      apiKeyPrefix: getSafePublicKeyPrefix(publicApiKey),
+      durationMs: getRequestDurationMs(req),
+      httpStatus: httpStatus.CREATED,
+      network: quote.network,
+      operation: "quote_create_best",
+      partnerId: req.authenticatedPartner?.id || partnerId || null,
+      partnerName: req.authenticatedPartner?.name || publicKeyPartnerName || null,
+      paymentMethod: quote.paymentMethod,
+      quoteId: quote.id,
+      rampType,
+      requestId: req.requestId,
+      status: "success",
+      userId: req.userId || null
+    });
+
     res.status(httpStatus.CREATED).json(quote);
   } catch (error) {
-    logger.error(`Error creating best quote: ${error instanceof Error ? error.message : String(error)}`);
+    logger.error("Error creating best quote", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeQuoteFailure(req, "quote_create_best", error, {
+      apiKeyPrefix: getSafePublicKeyPrefix(req.body?.apiKey || req.validatedPublicKey?.apiKey),
+      partnerId: req.authenticatedPartner?.id || req.body?.partnerId || null,
+      partnerName: req.authenticatedPartner?.name || req.validatedPublicKey?.partnerName || null,
+      rampType: req.body?.rampType
+    });
     next(error);
   }
 };
@@ -121,9 +170,68 @@ export const getQuote = async (
       });
     }
 
+    observeApiClientEvent({
+      durationMs: getRequestDurationMs(req),
+      httpStatus: httpStatus.OK,
+      network: quote.network,
+      operation: "quote_get",
+      paymentMethod: quote.paymentMethod,
+      quoteId: quote.id,
+      rampType: quote.rampType,
+      requestId: req.requestId,
+      status: "success",
+      userId: req.userId || null
+    });
+
     res.status(httpStatus.OK).json(quote);
   } catch (error) {
-    logger.error("Error getting quote:", error);
+    logger.error("Error getting quote", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeQuoteFailure(req, "quote_get", error, { quoteId: req.params.id });
     next(error);
   }
 };
+
+type QuoteOperation = "quote_create" | "quote_create_best" | "quote_get";
+
+interface ObservedQuoteRequest {
+  requestId?: string;
+  requestStartedAt?: number;
+  userId?: string;
+}
+
+function observeQuoteFailure(
+  req: ObservedQuoteRequest,
+  operation: QuoteOperation,
+  error: unknown,
+  context: {
+    apiKeyPrefix?: string | null;
+    network?: string | null;
+    partnerId?: string | null;
+    partnerName?: string | null;
+    paymentMethod?: string | null;
+    quoteId?: string | null;
+    rampType?: string | null;
+  } = {}
+): void {
+  const status = getHttpStatus(error);
+  observeApiClientEvent({
+    ...context,
+    durationMs: getRequestDurationMs(req),
+    errorMessage: getErrorMessage(error),
+    errorType: classifyApiClientError(error, status),
+    httpStatus: status,
+    operation,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
+function getHttpStatus(error: unknown): number {
+  return error instanceof APIError ? error.status || httpStatus.INTERNAL_SERVER_ERROR : httpStatus.INTERNAL_SERVER_ERROR;
+}
+
+function getSafePublicKeyPrefix(apiKey: string | null | undefined): string | null {
+  if (!apiKey?.startsWith("pk_")) return null;
+  return apiKey.slice(0, 8);
+}
diff --git a/apps/api/src/api/controllers/ramp.controller.ts b/apps/api/src/api/controllers/ramp.controller.ts
index 63bf81294..21f1579bc 100644
--- a/apps/api/src/api/controllers/ramp.controller.ts
+++ b/apps/api/src/api/controllers/ramp.controller.ts
@@ -17,6 +17,10 @@ import logger from "../../config/logger";
 import { APIError } from "../errors/api-error";
 import { enrichAdditionalDataWithClientIp } from "../helpers/clientIp";
 import { assertQuoteOwnership, assertRampOwnership } from "../middlewares/ownershipAuth";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { classifyApiClientError, getErrorMessage } from "../observability/errorClassifier";
+import { getRequestDurationMs } from "../observability/requestContext";
+import { ApiClientOperation } from "../observability/types";
 import rampService from "../services/ramp/ramp.service";
 
 /**
@@ -46,9 +50,17 @@ export const registerRamp = async (req: Request, res: Response<RampProcess>, nex
       userId: req.userId
     });
 
+    observeRampSuccess(req, "ramp_register", httpStatus.CREATED, {
+      paymentMethod: ramp.paymentMethod,
+      quoteId,
+      rampId: ramp.id,
+      rampType: ramp.type
+    });
+
     res.status(httpStatus.CREATED).json(ramp);
   } catch (error) {
-    logger.error("Error registering ramp:", error);
+    logger.error("Error registering ramp", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeRampFailure(req, "ramp_register", error, { quoteId: req.body?.quoteId || null });
     next(error);
   }
 };
@@ -90,9 +102,17 @@ export const updateRamp = async (
       rampId
     });
 
+    observeRampSuccess(req, "ramp_update", httpStatus.OK, {
+      paymentMethod: ramp.paymentMethod,
+      quoteId: ramp.quoteId,
+      rampId,
+      rampType: ramp.type
+    });
+
     res.status(httpStatus.OK).json(ramp);
   } catch (error) {
-    logger.error("Error updating ramp:", error);
+    logger.error("Error updating ramp", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeRampFailure(req, "ramp_update", error, { rampId: req.body?.rampId || null });
     next(error);
   }
 };
@@ -124,9 +144,17 @@ export const startRamp = async (
       rampId
     });
 
+    observeRampSuccess(req, "ramp_start", httpStatus.OK, {
+      paymentMethod: ramp.paymentMethod,
+      quoteId: ramp.quoteId,
+      rampId,
+      rampType: ramp.type
+    });
+
     res.status(httpStatus.OK).json(ramp);
   } catch (error) {
-    logger.error("Error starting ramp:", error);
+    logger.error("Error starting ramp", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeRampFailure(req, "ramp_start", error, { rampId: req.body?.rampId || null });
     next(error);
   }
 };
@@ -155,9 +183,17 @@ export const getRampStatus = async (
       });
     }
 
+    observeRampSuccess(req, "ramp_status", httpStatus.OK, {
+      paymentMethod: ramp.paymentMethod,
+      quoteId: ramp.quoteId,
+      rampId: id,
+      rampType: ramp.type
+    });
+
     res.status(httpStatus.OK).json(ramp);
   } catch (error) {
-    logger.error("Error getting ramp status:", error);
+    logger.error("Error getting ramp status", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeRampFailure(req, "ramp_status", error, { rampId: req.params.id });
     next(error);
   }
 };
@@ -185,9 +221,12 @@ export const getErrorLogs = async (
       });
     }
 
+    observeRampSuccess(req, "ramp_errors", httpStatus.OK, { rampId: id });
+
     res.status(httpStatus.OK).json(errorLogs);
   } catch (error) {
-    logger.error("Error getting error logs:", error);
+    logger.error("Error getting error logs", { errorType: classifyApiClientError(error), requestId: req.requestId });
+    observeRampFailure(req, "ramp_errors", error, { rampId: req.params.id });
     next(error);
   }
 };
@@ -234,3 +273,67 @@ export const getRampHistory = async (
     next(error);
   }
 };
+
+type RampObservedOperation = Extract<
+  ApiClientOperation,
+  "ramp_register" | "ramp_update" | "ramp_start" | "ramp_status" | "ramp_errors"
+>;
+
+interface RampObservationContext {
+  paymentMethod?: string | null;
+  quoteId?: string | null;
+  rampId?: string | null;
+  rampType?: string | null;
+}
+
+interface ObservedRampRequest {
+  authenticatedPartner?: { id: string; name: string };
+  requestId?: string;
+  requestStartedAt?: number;
+  userId?: string;
+}
+
+function observeRampSuccess(
+  req: ObservedRampRequest,
+  operation: RampObservedOperation,
+  status: number,
+  context: RampObservationContext
+): void {
+  observeApiClientEvent({
+    ...context,
+    durationMs: getRequestDurationMs(req),
+    httpStatus: status,
+    operation,
+    partnerId: req.authenticatedPartner?.id || null,
+    partnerName: req.authenticatedPartner?.name || null,
+    requestId: req.requestId,
+    status: "success",
+    userId: req.userId || null
+  });
+}
+
+function observeRampFailure(
+  req: ObservedRampRequest,
+  operation: RampObservedOperation,
+  error: unknown,
+  context: RampObservationContext
+): void {
+  const status = getHttpStatus(error);
+  observeApiClientEvent({
+    ...context,
+    durationMs: getRequestDurationMs(req),
+    errorMessage: getErrorMessage(error),
+    errorType: classifyApiClientError(error, status),
+    httpStatus: status,
+    operation,
+    partnerId: req.authenticatedPartner?.id || null,
+    partnerName: req.authenticatedPartner?.name || null,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
+function getHttpStatus(error: unknown): number {
+  return error instanceof APIError ? error.status || httpStatus.INTERNAL_SERVER_ERROR : httpStatus.INTERNAL_SERVER_ERROR;
+}
diff --git a/apps/api/src/api/middlewares/apiKeyAuth.ts b/apps/api/src/api/middlewares/apiKeyAuth.ts
index 276cb11a6..adb15bde3 100644
--- a/apps/api/src/api/middlewares/apiKeyAuth.ts
+++ b/apps/api/src/api/middlewares/apiKeyAuth.ts
@@ -1,10 +1,14 @@
 import { NextFunction, Request, Response } from "express";
 import logger from "../../config/logger";
 import Partner from "../../models/partner.model";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { getRequestDurationMs } from "../observability/requestContext";
+import { ApiClientErrorType } from "../observability/types";
 import { AuthenticatedPartner, getKeyType, isValidSecretKeyFormat, validateApiKey } from "./apiKeyAuth.helpers";
 
 // Extend Express Request type to include authenticatedPartner
 declare global {
+  // biome-ignore lint/style/noNamespace: Express request augmentation follows the existing backend pattern.
   namespace Express {
     interface Request {
       authenticatedPartner?: AuthenticatedPartner;
@@ -31,6 +35,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
       // No API key provided
       if (!apiKey) {
         if (options.required) {
+          recordAuthFailure(req, 401, "auth_missing_api_key");
           return res.status(401).json({
             error: {
               code: "API_KEY_REQUIRED",
@@ -46,6 +51,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
       // Validate that it's a secret key format (sk_*)
       const keyType = getKeyType(apiKey);
       if (keyType !== "secret") {
+        recordAuthFailure(req, 401, "auth_invalid_api_key", getSafeKeyPrefix(apiKey));
         return res.status(401).json({
           error: {
             code: "INVALID_SECRET_KEY",
@@ -57,6 +63,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
       }
 
       if (!isValidSecretKeyFormat(apiKey)) {
+        recordAuthFailure(req, 401, "auth_invalid_api_key", getSafeKeyPrefix(apiKey));
         return res.status(401).json({
           error: {
             code: "INVALID_SECRET_KEY_FORMAT",
@@ -70,6 +77,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
       const partner = await validateApiKey(apiKey);
 
       if (!partner) {
+        recordAuthFailure(req, 401, "auth_invalid_api_key", getSafeKeyPrefix(apiKey));
         return res.status(401).json({
           error: {
             code: "INVALID_API_KEY",
@@ -96,6 +104,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
           const requestedPartner = await Partner.findByPk(partnerIdOrName);
 
           if (!requestedPartner) {
+            recordAuthFailure(req, 404, "auth_invalid_public_key", getSafeKeyPrefix(apiKey), partner);
             return res.status(404).json({
               error: {
                 code: "PARTNER_NOT_FOUND",
@@ -113,6 +122,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
 
         // Compare partner names since one API key works for all partners with same name
         if (requestedPartnerName !== partner.name) {
+          recordAuthFailure(req, 403, "auth_partner_mismatch", getSafeKeyPrefix(apiKey), partner);
           return res.status(403).json({
             error: {
               code: "PARTNER_MISMATCH",
@@ -148,6 +158,7 @@ export function enforcePartnerAuth() {
     if (req.body?.partnerId) {
       // Partner must be authenticated
       if (!req.authenticatedPartner) {
+        recordAuthFailure(req, 403, "auth_missing_api_key");
         return res.status(403).json({
           error: {
             code: "AUTHENTICATION_REQUIRED",
@@ -169,6 +180,7 @@ export function enforcePartnerAuth() {
         const requestedPartner = await Partner.findByPk(partnerIdOrName);
 
         if (!requestedPartner) {
+          recordAuthFailure(req, 404, "auth_invalid_public_key", null, req.authenticatedPartner);
           return res.status(404).json({
             error: {
               code: "PARTNER_NOT_FOUND",
@@ -186,6 +198,7 @@ export function enforcePartnerAuth() {
 
       // Compare partner names (not IDs) since one API key works for all partners with same name
       if (requestedPartnerName !== req.authenticatedPartner.name) {
+        recordAuthFailure(req, 403, "auth_partner_mismatch", null, req.authenticatedPartner);
         return res.status(403).json({
           error: {
             code: "PARTNER_MISMATCH",
@@ -203,3 +216,33 @@ export function enforcePartnerAuth() {
     next();
   };
 }
+
+function recordAuthFailure(
+  req: Request,
+  httpStatus: number,
+  errorType: Extract<
+    ApiClientErrorType,
+    "auth_missing_api_key" | "auth_invalid_api_key" | "auth_invalid_public_key" | "auth_partner_mismatch"
+  >,
+  apiKeyPrefix?: string | null,
+  partner?: AuthenticatedPartner
+): void {
+  observeApiClientEvent({
+    apiKeyPrefix,
+    durationMs: getRequestDurationMs(req),
+    errorType,
+    httpStatus,
+    operation: "auth_api_key",
+    partnerId: partner?.id || req.authenticatedPartner?.id || null,
+    partnerName: partner?.name || req.authenticatedPartner?.name || null,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
+function getSafeKeyPrefix(apiKey: string | undefined): string | null {
+  if (!apiKey) return null;
+  if (!apiKey.startsWith("pk_") && !apiKey.startsWith("sk_")) return null;
+  return apiKey.slice(0, 8);
+}
diff --git a/apps/api/src/api/middlewares/dualAuth.test.ts b/apps/api/src/api/middlewares/dualAuth.test.ts
index 97d2a31e5..6b682c0c2 100644
--- a/apps/api/src/api/middlewares/dualAuth.test.ts
+++ b/apps/api/src/api/middlewares/dualAuth.test.ts
@@ -49,6 +49,7 @@ describe("assertQuoteOwnership", () => {
     })) as typeof QuoteTicket.findByPk;
     Partner.findByPk = mock(async () => ({
       id: "quote-partner-id",
+      isActive: true,
       name: "Partner"
     })) as typeof Partner.findByPk;
 
diff --git a/apps/api/src/api/middlewares/dualAuth.ts b/apps/api/src/api/middlewares/dualAuth.ts
index a7ca0e554..cb83f13c0 100644
--- a/apps/api/src/api/middlewares/dualAuth.ts
+++ b/apps/api/src/api/middlewares/dualAuth.ts
@@ -1,5 +1,7 @@
 import { NextFunction, Request, Response } from "express";
 import logger from "../../config/logger";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { getRequestDurationMs } from "../observability/requestContext";
 import { SupabaseAuthService } from "../services/auth";
 import { getKeyType, isValidSecretKeyFormat, validateSecretApiKey } from "./apiKeyAuth.helpers";
 
@@ -35,6 +37,7 @@ function dualAuthHandler({ requireCredentials }: { requireCredentials: boolean }
       if (apiKey) {
         const keyType = getKeyType(apiKey);
         if (keyType !== "secret" || !isValidSecretKeyFormat(apiKey)) {
+          recordDualAuthFailure(req, 401, "auth_invalid_api_key", getSafeKeyPrefix(apiKey));
           return res.status(401).json({
             error: {
               code: "INVALID_SECRET_KEY",
@@ -46,6 +49,7 @@ function dualAuthHandler({ requireCredentials }: { requireCredentials: boolean }
 
         const partner = await validateSecretApiKey(apiKey);
         if (!partner) {
+          recordDualAuthFailure(req, 401, "auth_invalid_api_key", getSafeKeyPrefix(apiKey));
           return res.status(401).json({
             error: {
               code: "INVALID_API_KEY",
@@ -63,6 +67,7 @@ function dualAuthHandler({ requireCredentials }: { requireCredentials: boolean }
         const token = authHeader.slice(7);
         const result = await SupabaseAuthService.verifyToken(token);
         if (!result.valid) {
+          recordDualAuthFailure(req, 401, "auth_invalid_api_key");
           return res.status(401).json({
             error: {
               code: "INVALID_BEARER_TOKEN",
@@ -81,6 +86,7 @@ function dualAuthHandler({ requireCredentials }: { requireCredentials: boolean }
         return next();
       }
 
+      recordDualAuthFailure(req, 401, "auth_missing_api_key");
       return res.status(401).json({
         error: {
           code: "AUTHENTICATION_REQUIRED",
@@ -94,3 +100,28 @@ function dualAuthHandler({ requireCredentials }: { requireCredentials: boolean }
     }
   };
 }
+
+function recordDualAuthFailure(
+  req: Request,
+  httpStatus: number,
+  errorType: "auth_missing_api_key" | "auth_invalid_api_key",
+  apiKeyPrefix?: string | null
+): void {
+  observeApiClientEvent({
+    apiKeyPrefix,
+    durationMs: getRequestDurationMs(req),
+    errorType,
+    httpStatus,
+    operation: "auth_dual",
+    partnerId: req.authenticatedPartner?.id || null,
+    partnerName: req.authenticatedPartner?.name || null,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
+function getSafeKeyPrefix(apiKey: string | undefined): string | null {
+  if (!apiKey?.startsWith("sk_")) return null;
+  return apiKey.slice(0, 8);
+}
diff --git a/apps/api/src/api/middlewares/ownershipAuth.ts b/apps/api/src/api/middlewares/ownershipAuth.ts
index e4fc5ff27..97b75d512 100644
--- a/apps/api/src/api/middlewares/ownershipAuth.ts
+++ b/apps/api/src/api/middlewares/ownershipAuth.ts
@@ -1,11 +1,19 @@
-import { Request } from "express";
 import httpStatus from "http-status";
 import Partner from "../../models/partner.model";
 import QuoteTicket from "../../models/quoteTicket.model";
 import RampState from "../../models/rampState.model";
 import { APIError } from "../errors/api-error";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { getRequestDurationMs } from "../observability/requestContext";
 import type { AuthenticatedPartner } from "./apiKeyAuth.helpers";
 
+interface OwnershipRequest {
+  authenticatedPartner?: AuthenticatedPartner;
+  requestId?: string;
+  requestStartedAt?: number;
+  userId?: string;
+}
+
 async function ownsPartnerRecord(authenticatedPartner: AuthenticatedPartner, partnerId: string | null): Promise<boolean> {
   if (!partnerId) {
     return false;
@@ -23,21 +31,21 @@ async function ownsPartnerRecord(authenticatedPartner: AuthenticatedPartner, par
  * or req.body.rampId. Partner principals must match the quote's partnerId;
  * user principals must match the ramp state's userId.
  */
-export async function assertRampOwnership(
-  req: Pick<Request, "authenticatedPartner" | "userId">,
-  rampId: string
-): Promise<void> {
+export async function assertRampOwnership(req: OwnershipRequest, rampId: string): Promise<void> {
   const ramp = await RampState.findByPk(rampId);
   if (!ramp) {
+    recordOwnershipFailure(req, httpStatus.NOT_FOUND, "ramp_not_found", { rampId });
     throw new APIError({ message: "Ramp not found", status: httpStatus.NOT_FOUND });
   }
 
   if (req.authenticatedPartner) {
     const quote = await QuoteTicket.findByPk(ramp.quoteId);
     if (!quote) {
+      recordOwnershipFailure(req, httpStatus.NOT_FOUND, "quote_not_found", { quoteId: ramp.quoteId, rampId });
       throw new APIError({ message: "Associated quote not found", status: httpStatus.NOT_FOUND });
     }
     if (!(await ownsPartnerRecord(req.authenticatedPartner, quote.partnerId))) {
+      recordOwnershipFailure(req, httpStatus.FORBIDDEN, "ownership_denied", { quoteId: ramp.quoteId, rampId });
       throw new APIError({
         message: "Authenticated partner does not own this ramp",
         status: httpStatus.FORBIDDEN
@@ -48,6 +56,7 @@ export async function assertRampOwnership(
 
   if (req.userId) {
     if (ramp.userId !== req.userId) {
+      recordOwnershipFailure(req, httpStatus.FORBIDDEN, "ownership_denied", { quoteId: ramp.quoteId, rampId });
       throw new APIError({
         message: "Authenticated user does not own this ramp",
         status: httpStatus.FORBIDDEN
@@ -62,6 +71,7 @@ export async function assertRampOwnership(
   if (ramp.userId === null) {
     const quote = await QuoteTicket.findByPk(ramp.quoteId);
     if (!quote) {
+      recordOwnershipFailure(req, httpStatus.NOT_FOUND, "quote_not_found", { quoteId: ramp.quoteId, rampId });
       throw new APIError({ message: "Associated quote not found", status: httpStatus.NOT_FOUND });
     }
     if (quote.partnerId === null) {
@@ -69,23 +79,23 @@ export async function assertRampOwnership(
     }
   }
 
+  recordOwnershipFailure(req, httpStatus.UNAUTHORIZED, "ownership_denied", { quoteId: ramp.quoteId, rampId });
   throw new APIError({ message: "Authentication required", status: httpStatus.UNAUTHORIZED });
 }
 
 /**
  * Ownership check for flows that reference a quote before a ramp exists.
  */
-export async function assertQuoteOwnership(
-  req: Pick<Request, "authenticatedPartner" | "userId">,
-  quoteId: string
-): Promise<void> {
+export async function assertQuoteOwnership(req: OwnershipRequest, quoteId: string): Promise<void> {
   const quote = await QuoteTicket.findByPk(quoteId);
   if (!quote) {
+    recordOwnershipFailure(req, httpStatus.NOT_FOUND, "quote_not_found", { quoteId });
     throw new APIError({ message: "Quote not found", status: httpStatus.NOT_FOUND });
   }
 
   if (req.authenticatedPartner) {
     if (!(await ownsPartnerRecord(req.authenticatedPartner, quote.partnerId))) {
+      recordOwnershipFailure(req, httpStatus.FORBIDDEN, "ownership_denied", { quoteId });
       throw new APIError({
         message: "Authenticated partner does not own this quote",
         status: httpStatus.FORBIDDEN
@@ -96,12 +106,14 @@ export async function assertQuoteOwnership(
 
   if (req.userId) {
     if (quote.partnerId !== null) {
+      recordOwnershipFailure(req, httpStatus.FORBIDDEN, "ownership_denied", { quoteId });
       throw new APIError({
         message: "This quote belongs to a partner; user authentication is not sufficient",
         status: httpStatus.FORBIDDEN
       });
     }
     if (quote.userId !== null && quote.userId !== req.userId) {
+      recordOwnershipFailure(req, httpStatus.FORBIDDEN, "ownership_denied", { quoteId });
       throw new APIError({
         message: "Authenticated user does not own this quote",
         status: httpStatus.FORBIDDEN
@@ -117,5 +129,26 @@ export async function assertQuoteOwnership(
     return;
   }
 
+  recordOwnershipFailure(req, httpStatus.UNAUTHORIZED, "ownership_denied", { quoteId });
   throw new APIError({ message: "Authentication required", status: httpStatus.UNAUTHORIZED });
 }
+
+function recordOwnershipFailure(
+  req: OwnershipRequest,
+  status: number,
+  errorType: "ownership_denied" | "quote_not_found" | "ramp_not_found",
+  context: { quoteId?: string | null; rampId?: string | null }
+): void {
+  observeApiClientEvent({
+    ...context,
+    durationMs: getRequestDurationMs(req),
+    errorType,
+    httpStatus: status,
+    operation: "auth_ownership",
+    partnerId: req.authenticatedPartner?.id || null,
+    partnerName: req.authenticatedPartner?.name || null,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
diff --git a/apps/api/src/api/middlewares/publicKeyAuth.ts b/apps/api/src/api/middlewares/publicKeyAuth.ts
index bb581f391..57aee5a0b 100644
--- a/apps/api/src/api/middlewares/publicKeyAuth.ts
+++ b/apps/api/src/api/middlewares/publicKeyAuth.ts
@@ -1,9 +1,12 @@
 import { NextFunction, Request, Response } from "express";
 import logger from "../../config/logger";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { getRequestDurationMs } from "../observability/requestContext";
 import { getKeyType, isValidApiKeyFormat, validatePublicApiKey } from "./apiKeyAuth.helpers";
 
 // Extend Express Request type to include validated public key
 declare global {
+  // biome-ignore lint/style/noNamespace: Express request augmentation follows the existing backend pattern.
   namespace Express {
     interface Request {
       validatedPublicKey?: {
@@ -33,6 +36,7 @@ export function validatePublicKey() {
 
       // Validate API key format
       if (!isValidApiKeyFormat(apiKey)) {
+        recordPublicKeyFailure(req, 400, getSafeKeyPrefix(apiKey));
         return res.status(400).json({
           error: {
             code: "INVALID_API_KEY_FORMAT",
@@ -45,6 +49,7 @@ export function validatePublicKey() {
       // Check if it's a public key
       const keyType = getKeyType(apiKey);
       if (keyType !== "public") {
+        recordPublicKeyFailure(req, 400, getSafeKeyPrefix(apiKey));
         return res.status(400).json({
           error: {
             code: "INVALID_KEY_TYPE",
@@ -58,6 +63,7 @@ export function validatePublicKey() {
       const partnerName = await validatePublicApiKey(apiKey);
 
       if (!partnerName) {
+        recordPublicKeyFailure(req, 401, getSafeKeyPrefix(apiKey));
         return res.status(401).json({
           error: {
             code: "INVALID_PUBLIC_KEY",
@@ -80,3 +86,22 @@ export function validatePublicKey() {
     }
   };
 }
+
+function recordPublicKeyFailure(req: Request, httpStatus: number, apiKeyPrefix: string | null): void {
+  observeApiClientEvent({
+    apiKeyPrefix,
+    durationMs: getRequestDurationMs(req),
+    errorType: "auth_invalid_public_key",
+    httpStatus,
+    operation: "auth_public_key",
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
+function getSafeKeyPrefix(apiKey: string | undefined): string | null {
+  if (!apiKey) return null;
+  if (!apiKey.startsWith("pk_") && !apiKey.startsWith("sk_")) return null;
+  return apiKey.slice(0, 8);
+}
diff --git a/apps/api/src/api/middlewares/validators.ts b/apps/api/src/api/middlewares/validators.ts
index 95c947cc1..8924973e6 100644
--- a/apps/api/src/api/middlewares/validators.ts
+++ b/apps/api/src/api/middlewares/validators.ts
@@ -29,6 +29,8 @@ import { CONTACT_SHEET_HEADER_VALUES } from "../controllers/contact.controller";
 import { EMAIL_SHEET_HEADER_VALUES } from "../controllers/email.controller";
 import { RATING_SHEET_HEADER_VALUES } from "../controllers/rating.controller";
 import { FLOW_HEADERS } from "../controllers/storage.controller";
+import { observeApiClientEvent } from "../observability/apiClientEvent.service";
+import { getRequestDurationMs } from "../observability/requestContext";
 
 interface CreationBody {
   accountId: string;
@@ -380,16 +382,19 @@ export const validateCreateQuoteInput: RequestHandler<unknown, unknown, CreateQu
   const { rampType, from, to, inputAmount, inputCurrency, outputCurrency } = req.body;
 
   if (!rampType || !from || !to || !inputAmount || !inputCurrency || !outputCurrency) {
+    observeQuoteValidationFailure(req, "quote_create");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.MissingRequiredFields });
     return;
   }
 
   if (rampType !== RampDirection.BUY && rampType !== RampDirection.SELL) {
+    observeQuoteValidationFailure(req, "quote_create");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.InvalidRampType });
     return;
   }
 
   if (!validateSupportedFiatCurrency(rampType, inputCurrency, outputCurrency, res)) {
+    observeQuoteValidationFailure(req, "quote_create");
     return;
   }
 
@@ -405,38 +410,45 @@ export const validateCreateBestQuoteInput: RequestHandler<unknown, unknown, Crea
   const { rampType, from, to, inputAmount, inputCurrency, outputCurrency, networks } = req.body;
 
   if (!rampType || !inputAmount || !inputCurrency || !outputCurrency) {
+    observeQuoteValidationFailure(req, "quote_create_best");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.MissingRequiredFields });
     return;
   }
 
   if (rampType !== RampDirection.BUY && rampType !== RampDirection.SELL) {
+    observeQuoteValidationFailure(req, "quote_create_best");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.InvalidRampType });
     return;
   }
 
   if (rampType === RampDirection.BUY && !from) {
+    observeQuoteValidationFailure(req, "quote_create_best");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.MissingFromField });
     return;
   }
 
   if (rampType === RampDirection.SELL && !to) {
+    observeQuoteValidationFailure(req, "quote_create_best");
     res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.MissingToField });
     return;
   }
 
   if (networks !== undefined) {
     if (!Array.isArray(networks)) {
+      observeQuoteValidationFailure(req, "quote_create_best");
       res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.InvalidNetworks });
       return;
     }
     const normalized: Networks[] = [];
     for (const entry of networks) {
       if (typeof entry !== "string") {
+        observeQuoteValidationFailure(req, "quote_create_best");
         res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.InvalidNetworks });
         return;
       }
       const canonical = getCaseSensitiveNetwork(entry);
       if (!canonical) {
+        observeQuoteValidationFailure(req, "quote_create_best");
         res.status(httpStatus.BAD_REQUEST).json({ message: QuoteError.InvalidNetworks });
         return;
       }
@@ -446,6 +458,7 @@ export const validateCreateBestQuoteInput: RequestHandler<unknown, unknown, Crea
   }
 
   if (!validateSupportedFiatCurrency(rampType, inputCurrency, outputCurrency, res)) {
+    observeQuoteValidationFailure(req, "quote_create_best");
     return;
   }
 
@@ -458,6 +471,21 @@ const normalizeAxlUsdcCurrency = (value: unknown): unknown => {
   return value.toLowerCase() === "axlusdc" ? "USDC.axl" : value;
 };
 
+function observeQuoteValidationFailure(
+  req: { requestId?: string; requestStartedAt?: number; userId?: string },
+  operation: "quote_create" | "quote_create_best"
+): void {
+  observeApiClientEvent({
+    durationMs: getRequestDurationMs(req),
+    errorType: "validation_error",
+    httpStatus: httpStatus.BAD_REQUEST,
+    operation,
+    requestId: req.requestId,
+    status: "failure",
+    userId: req.userId || null
+  });
+}
+
 export const validateGetWidgetUrlInput: RequestHandler<unknown, unknown, GetWidgetUrlLocked | GetWidgetUrlRefresh> = (
   req,
   res,

From 42c45b3a1868da33de5057f71e1a34f8cd5afc5f Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 09:50:15 +0200
Subject: [PATCH 04/11] Document client observability security controls

---
 .../07-operations/api-surface.md              | 18 +++++--
 .../07-operations/client-observability.md     | 49 +++++++++++++++++++
 .../07-operations/secret-management.md        |  3 ++
 docs/security-spec/README.md                  |  3 ++
 4 files changed, 69 insertions(+), 4 deletions(-)
 create mode 100644 docs/security-spec/07-operations/client-observability.md

diff --git a/docs/security-spec/07-operations/api-surface.md b/docs/security-spec/07-operations/api-surface.md
index 3037e381c..56c1a6177 100644
--- a/docs/security-spec/07-operations/api-surface.md
+++ b/docs/security-spec/07-operations/api-surface.md
@@ -8,7 +8,7 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 - CORS: Explicit origin whitelist — `app.vortexfinance.co`, `metrics.vortexfinance.co`, staging Netlify, `localhost` (dev only)
 - Rate limiting: 100 requests per minute per IP (global, all endpoints)
 - Helmet: Standard HTTP security headers
-- Body parser: JSON with **50MB limit**
+- Body parser: JSON with **20MB limit**
 - Cookie parser: Enabled (for Supabase auth tokens)
 
 **Input validation** (`middlewares/validators.ts`):
@@ -22,25 +22,32 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 - 404 handler for unmatched routes
 - Error responses include an `errors` array with validation details
 
+**Request correlation and client observability** (`api/observability/`):
+- Incoming requests receive or propagate a non-secret request ID.
+- The API returns `X-Request-ID` so clients can include it in support/debug reports.
+- Partner-facing quote/ramp/auth outcomes are recorded as sanitized operational events; see `07-operations/client-observability.md`.
+
 **Route structure:** 27 route files under `api/routes/v1/`, each mounting controllers with appropriate auth middleware.
 
 ## Security Invariants
 
 1. **CORS MUST only allow explicit origins** — The whitelist is defined in `express.ts`. No wildcard (`*`) origins. No dynamic origin reflection (echoing back the `Origin` header).
 2. **Rate limiting MUST be enforced on all endpoints** — 100 req/min per IP applies globally via `express-rate-limit`. No endpoint should bypass this.
-3. **Body size MUST be bounded** — The JSON body parser has a limit. **⚠️ FINDING: The limit is 50MB (`"50mb"`), which is excessively large for a JSON API.** A typical API allows 1-10MB. 50MB enables memory exhaustion attacks.
+3. **Body size MUST be bounded** — The JSON body parser has a limit. **⚠️ FINDING: The limit is 20MB (`"20mb"`), which is still large for a JSON API.** A typical API allows 1-10MB. 20MB still enables avoidable memory pressure.
 4. **All user input MUST be validated before reaching controllers** — Validators run as middleware before the controller function. Missing validation on an endpoint means raw user input reaches business logic.
 5. **Error responses MUST NOT leak internal details in production** — Stack traces are stripped when `NODE_ENV !== "development"`. Error messages should be generic. The `errors` array should contain only user-facing validation messages.
 6. **404 responses MUST be returned for unmatched routes** — The 404 handler prevents Express from returning default HTML error pages that could reveal framework information.
 7. **Helmet MUST be enabled** — Adds `X-Frame-Options`, `X-Content-Type-Options`, `Strict-Transport-Security`, and other security headers.
 8. **Input validation MUST cover all mutable endpoints** — Every POST/PUT/PATCH/DELETE endpoint should have a validator middleware. GET endpoints with query parameters should also validate.
 9. **No endpoint MUST accept and process fields not explicitly validated** — Hand-written validators check specific fields but don't reject unknown fields. Extra fields pass through to controllers, which could lead to mass assignment or unexpected behavior.
+10. **Request IDs MUST be correlation-only** — Request IDs may be accepted from clients or generated by the API, but they must not grant access, alter authorization, or be treated as trusted identity.
+11. **API observability MUST NOT change request outcomes** — Client event persistence/logging must be best-effort and must not change controller response bodies, status codes, or ramp/quote state.
 
 ## Threat Vectors & Mitigations
 
 | Threat | Mitigation |
 |---|---|
-| **⚠️ Memory exhaustion via large request body** — Attacker sends a 50MB JSON payload repeatedly to exhaust server memory | Rate limiting (100 req/min) provides some protection, but 100 requests × 50MB = 5GB of memory pressure per minute per IP. **The 50MB limit should be reduced to 1-10MB.** |
+| **⚠️ Memory exhaustion via large request body** — Attacker sends a 20MB JSON payload repeatedly to exhaust server memory | Rate limiting (100 req/min) provides some protection, but 100 requests × 20MB = 2GB of memory pressure per minute per IP. **The 20MB limit should be reduced to 1-10MB.** |
 | **CORS bypass** — Attacker's site makes cross-origin requests to the API | Explicit origin whitelist prevents this. However, the whitelist includes `staging--pendulum-pay.netlify.app` — if the staging site is compromised or has XSS, it becomes a CORS-allowed origin in production. |
 | **Rate limit bypass via IP rotation** — Attacker uses multiple IPs to exceed per-IP rate limits | No mitigation beyond the per-IP limit. No account-based rate limiting, no endpoint-specific limits, no progressive penalties. High-value endpoints (ramp creation, quote generation) get the same limit as read-only endpoints. |
 | **Input validation bypass** — Validator doesn't check a field that the controller uses | Hand-written validators are prone to omissions. No schema library enforces completeness. New fields added to controllers may not get corresponding validators. |
@@ -49,10 +56,11 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 | **Staging CORS origin in production** — `staging--pendulum-pay.netlify.app` is in the CORS whitelist | If the staging site has an XSS vulnerability, an attacker could use it to make authenticated cross-origin requests to the production API. Staging origins should ideally be removed from production CORS config. |
 | **No per-endpoint rate limiting** — Sensitive endpoints (ramp creation, admin operations) have the same rate limit as public read endpoints | An attacker can create 100 ramps per minute per IP. For endpoints that trigger expensive operations (XCM, SquidRouter), this could amplify costs. |
 | **Cookie-based auth without CSRF protection** — Cookie parser is enabled for Supabase auth tokens | If auth tokens are stored in cookies (not just headers), cross-site requests from CORS-allowed origins could carry auth cookies automatically. Verify whether CSRF tokens or `SameSite` cookie attributes are used. |
+| **Observability side effects** — Event persistence failure breaks a partner-facing API call | Observability helpers must catch persistence/logging errors and run best-effort only. See `client-observability.md`. |
 
 ## Audit Checklist
 
-- [FAIL] **⚠️ FINDING F-035**: `bodyParser.json({ limit: "50mb" })` — verify this limit is intentional. Recommend reducing to 1-10MB for a JSON API. **FAIL F-035** — 50MB limit is excessive; enables memory exhaustion attacks.
+- [FAIL] **⚠️ FINDING F-035**: `bodyParser.json({ limit: "20mb" })` — verify this limit is intentional. Recommend reducing to 1-10MB for a JSON API. **FAIL F-035** — 20MB limit remains high for a JSON API.
 - [FAIL] **FINDING F-036**: `staging--pendulum-pay.netlify.app` is in the production CORS whitelist — verify this is intentional and assess the risk of staging-site compromise. **FAIL F-036** — staging origin always in CORS whitelist regardless of `NODE_ENV`.
 - [PARTIAL] **FINDING**: All validators are hand-written (no Zod/Joi) — verify every mutable endpoint has a corresponding validator middleware. **PARTIAL F-037** — hand-written validators exist but multiple sensitive endpoints lack authentication/validation entirely.
 - [x] Verify CORS does not use wildcard (`*`) or dynamic origin reflection — check `express.ts` for `origin: true` or callback patterns. **PASS** — explicit origin whitelist used; no wildcard or dynamic reflection.
@@ -68,3 +76,5 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 - [PARTIAL] Check whether Supabase auth cookies use `SameSite=Strict` or `SameSite=Lax` — and whether CSRF tokens are required for state-changing operations. **PARTIAL** — cookie parser enabled but cookie attributes not explicitly configured for `SameSite`.
 - [x] Verify the 404 handler does not reveal Express version or framework information. **PASS** — custom 404 handler returns generic JSON error.
 - [x] Check all 27 route files for endpoints that accept file uploads — verify file size limits and type validation if present. **PASS** — no file upload endpoints found.
+- [ ] Verify request ID middleware runs before routes and returns `X-Request-ID` without using request IDs for authorization.
+- [ ] Verify partner-facing API observability writes are best-effort and cannot alter response status, response body, or quote/ramp state.
diff --git a/docs/security-spec/07-operations/client-observability.md b/docs/security-spec/07-operations/client-observability.md
new file mode 100644
index 000000000..73e11ae36
--- /dev/null
+++ b/docs/security-spec/07-operations/client-observability.md
@@ -0,0 +1,49 @@
+# Client Observability
+
+## What This Does
+
+Backend client observability records sanitized operational events for partner-facing API activity. It is designed to help operators identify when one API client or partner integration is having problems without changing quote, ramp, authentication, or phase-processing behavior.
+
+The observed surface includes:
+
+- API key, public key, dual-auth, and ownership failures.
+- Quote create, best-quote create, and quote retrieval.
+- Ramp register, update, start, status, and error-log retrieval.
+- Request correlation through `X-Request-ID` / `X-Correlation-ID` and response `X-Request-ID`.
+
+Events are persisted in `api_client_events` and structured logs are emitted through the existing backend logger. The event table is an operational telemetry store, not a source of truth for ramp state. Ramp execution failures remain in `RampState.errorLogs`; client observability events are request-level records used for dashboards, alerting, and incident investigation.
+
+## Security Invariants
+
+1. **Observability MUST NOT affect API behavior** — Event persistence, structured logging, and metric hooks must be best-effort. Failures in the observability layer must not change response bodies, HTTP statuses, ramp state, quote state, or retry behavior.
+2. **Events MUST be sanitized before persistence** — Only approved scalar fields may be stored. Raw request bodies, raw headers, nested metadata objects, and sensitive keys must be dropped before inserting `api_client_events` rows.
+3. **Secrets MUST NOT be logged or persisted** — `X-API-Key`, bearer tokens, secret API keys, provider credentials, private keys, seeds, ephemeral private material, and signed transaction payloads must not appear in logs or observability events.
+4. **Sensitive user/payment data MUST NOT be logged or persisted** — Tax IDs, PIX destinations, QR codes, KYC data, bank details, and raw payment credentials must be excluded from observability metadata.
+5. **Request correlation MUST be non-secret** — `requestId`, `quoteId`, and `rampId` may be stored for debugging, but they must not be used as high-cardinality metric labels. They are correlation identifiers, not authentication material.
+6. **Partner attribution MUST use safe identifiers** — Events may store `partnerId`, `partnerName`, and short API key prefixes. Full secret keys and raw auth headers are forbidden.
+7. **Operational metrics MUST remain low-cardinality** — Future metric exporters must group by bounded labels such as operation, partner, status, HTTP status, and error type. They must not label by user ID, wallet address, request ID, quote ID, ramp ID, tax ID, PIX key, or free-form request values.
+8. **Event persistence SHOULD have automated retention before production alerting/dashboard rollout** — Raw operational events are useful for investigation but should not be retained indefinitely without aggregation or cleanup. Until that follow-up exists, operators should treat retention as a known operational gap.
+
+## Threat Vectors & Mitigations
+
+| Threat | Mitigation |
+|---|---|
+| **Observability database leak** — An attacker gains read access to `api_client_events` | Store only minimal sanitized event fields. Do not persist secrets, raw request bodies, tax IDs, PIX data, KYC data, or private key material. Treat the table as operationally sensitive even after redaction. |
+| **API key/header capture** — Instrumentation accidentally records `X-API-Key`, bearer tokens, or raw headers | Use an allowlist-shaped event schema and denylist sensitive metadata keys before persistence. Store only short key prefixes when explicitly safe. |
+| **PII leakage through metadata** — Client-provided `additionalData` or error messages include tax IDs, PIX keys, or bank details | Do not persist nested metadata objects. Keep metadata scalar-only and sanitized. Avoid passing request bodies to observability helpers. Truncate error messages and prefer stable `errorType` categories. |
+| **Business flow disruption** — Database/logging outage causes quote/ramp requests to fail | Observability writes are fire-and-forget/best-effort and catch their own errors. The request path must proceed exactly as it would without observability. |
+| **Missing correlation during incidents** — Operators cannot connect a partner report to backend logs | Generate or propagate `requestId` for all requests and return it via `X-Request-ID`. Persist request IDs alongside quote/ramp IDs when available. |
+| **High-cardinality metric explosion** — Future dashboard metrics use ramp IDs or user IDs as labels | Keep high-cardinality identifiers in logs/event rows only. Export aggregate metrics using bounded labels. |
+| **Unbounded telemetry retention** — Raw event rows grow indefinitely | Known follow-up: add retention or aggregation before long-term production alerting/dashboard operation. Initial raw retention should be time-bounded, ideally 30-90 days. |
+
+## Audit Checklist
+
+- [ ] Verify `requestContext` assigns `requestId` and `requestStartedAt` before request logging and route handling.
+- [ ] Verify `X-Request-ID` is returned on API responses and incoming `X-Request-ID` / `X-Correlation-ID` values are treated only as correlation IDs.
+- [ ] Verify `api_client_events` stores only the approved fields: operation, status, HTTP status, error type/message, safe partner attribution, quote/ramp IDs, duration, and sanitized metadata.
+- [ ] Verify event persistence helpers catch their own errors and cannot throw into controller or middleware responses.
+- [ ] Verify auth, quote, and ramp request instrumentation does not alter existing response bodies or HTTP status codes.
+- [ ] Verify no observability event stores `X-API-Key`, bearer tokens, raw headers, raw request bodies, tax IDs, PIX destinations, QR codes, KYC data, private keys, seeds, ephemeral secrets, or signed transaction payloads.
+- [ ] Verify error messages are truncated and dashboards/alerts use stable `errorType` categories rather than raw messages.
+- [ ] Verify future metric exporters do not use request ID, quote ID, ramp ID, user ID, wallet address, tax ID, or PIX key as metric labels.
+- [ ] Add and verify an automated retention or aggregation mechanism before retaining high-volume production telemetry long-term.
diff --git a/docs/security-spec/07-operations/secret-management.md b/docs/security-spec/07-operations/secret-management.md
index 33abb4a53..efac1a7c9 100644
--- a/docs/security-spec/07-operations/secret-management.md
+++ b/docs/security-spec/07-operations/secret-management.md
@@ -55,6 +55,7 @@ This spec catalogs every secret, its purpose, its blast radius if compromised, a
 7. **Database credentials (`DB_*`) MUST NOT be accessible from the public internet** — Direct PostgreSQL access should be restricted to the application server's network.
 8. **No secret MUST be passed as a URL query parameter** — Query parameters are logged by proxies, CDNs, and web servers. Secrets must only travel in headers or request bodies.
 9. **`MYKOBO_CLIENT_DOMAIN` MUST be set in production** — Not a secret, but operationally critical: when unset, Mykobo silently applies its default fee tier (~5x worse than the negotiated rate). Quote-engine fee defaults will then diverge from what Mykobo actually charges. Deployment automation MUST treat a missing `MYKOBO_CLIENT_DOMAIN` as a hard failure rather than letting it fall through to default-tier fees.
+10. **Observability MUST follow the same no-secret rule as logs** — API client events, request correlation logs, metrics, and dashboard data must not contain full API keys, bearer tokens, provider credentials, private keys, seeds, raw request headers, or raw request bodies. See `07-operations/client-observability.md`.
 
 ## Threat Vectors & Mitigations
 
@@ -67,6 +68,7 @@ This spec catalogs every secret, its purpose, its blast radius if compromised, a
 | **Lateral movement from price provider keys** — Compromise of AlchemyPay/Transak/MoonPay keys | Limited blast radius — these keys access price data, not funds. However, an attacker could manipulate prices shown to users (if the provider API allows it) or access transaction data. |
 | **Google Sheets credentials** — Access to fee logging spreadsheet | Could expose fee data and ramp metadata. Could manipulate fee records. Lower severity than financial keys but still a data leak. |
 | **`SUPABASE_SERVICE_KEY` used for all database operations** — No principle of least privilege | The service key bypasses all RLS. If any code path leaks this key, the attacker has unrestricted database access. A more secure approach would use the anon key with RLS for read operations and the service key only for privileged writes. |
+| **Observability event leak** — Operational telemetry captures secret values or payment/KYC data | Client observability uses a sanitized event schema, short key prefixes only, scalar metadata filtering, and explicit exclusion of raw headers/bodies, tax IDs, PIX data, KYC data, and private material. |
 
 ## Audit Checklist
 
@@ -85,3 +87,4 @@ This spec catalogs every secret, its purpose, its blast radius if compromised, a
 - [x] Check whether `GOOGLE_PRIVATE_KEY` contains newlines that might be mis-parsed — a common issue with PEM keys in env vars. **PASS** — PEM key handling present; standard env var parsing.
 - [x] Map the full blast radius: if the API server is compromised, list every account, service, and database that becomes accessible. **PASS (comprehensive)** — full blast radius documented in the Secret Inventory table above.
 - [x] **FINDING F-062 (MEDIUM)**: Verify SDK does not log API keys or secrets to console. **PASS (FIXED)** — removed `console.log("Creating quote with request:", request)` from `ApiService.ts` that was leaking the full request object including API key.
+- [ ] Verify API client event persistence stores only short key prefixes and never stores full `X-API-Key`, bearer tokens, raw auth headers, or request bodies.
diff --git a/docs/security-spec/README.md b/docs/security-spec/README.md
index cd7f4437e..e620b8893 100644
--- a/docs/security-spec/README.md
+++ b/docs/security-spec/README.md
@@ -45,6 +45,7 @@ This directory contains the security specification for the Vortex cross-border p
 | Rebalancer | `07-operations/rebalancer.md` | Automated liquidity management |
 | Secret Management | `07-operations/secret-management.md` | Env vars, rotation, blast radius |
 | API Surface | `07-operations/api-surface.md` | Rate limiting, CORS, input validation, error handling |
+| Client Observability | `07-operations/client-observability.md` | Request IDs, sanitized API client events, operational monitoring |
 
 ## Per-File Format
 
@@ -74,3 +75,5 @@ Every spec file uses exactly four sections:
 | **pk\_/sk\_** | Public key / Secret key prefixes for the dual API key system |
 | **PIX** | Brazilian instant payment system |
 | **SEPA** | Single Euro Payments Area — European bank transfer system |
+| **Request ID** | Non-secret correlation identifier generated or propagated by the API for log/event debugging |
+| **Client event** | Sanitized operational record of a partner-facing API request outcome |

From 41709f460dc9dd71c96f6f6e14e46a8d582c66af Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 18:21:25 +0200
Subject: [PATCH 05/11] Add protected metrics dashboard endpoint

---
 apps/api/.env.example                         |   4 +
 .../admin/apiClientEvents.controller.ts       | 157 ++++++++++++++++++
 .../api/middlewares/metricsDashboardAuth.ts   |  90 ++++++++++
 .../v1/admin/api-client-events.route.ts       |  17 ++
 apps/api/src/api/routes/v1/index.ts           |   7 +
 apps/api/src/config/vars.ts                   |   3 +
 6 files changed, 278 insertions(+)
 create mode 100644 apps/api/src/api/controllers/admin/apiClientEvents.controller.ts
 create mode 100644 apps/api/src/api/middlewares/metricsDashboardAuth.ts
 create mode 100644 apps/api/src/api/routes/v1/admin/api-client-events.route.ts

diff --git a/apps/api/.env.example b/apps/api/.env.example
index f2698d3ba..906c4fd22 100644
--- a/apps/api/.env.example
+++ b/apps/api/.env.example
@@ -12,6 +12,10 @@ SANDBOX_ENABLED=false
 # Example: openssl rand -base64 32
 ADMIN_SECRET=your-secure-admin-secret-here
 
+# Internal metrics dashboard authentication
+# Use a different secret than ADMIN_SECRET to reduce blast radius.
+METRICS_DASHBOARD_SECRET=your-secure-metrics-dashboard-secret-here
+
 # Supabase Configuration
 SUPABASE_URL=https://your-project-id.supabase.co
 SUPABASE_ANON_KEY=your-anon-key-here
diff --git a/apps/api/src/api/controllers/admin/apiClientEvents.controller.ts b/apps/api/src/api/controllers/admin/apiClientEvents.controller.ts
new file mode 100644
index 000000000..9da8c1f2e
--- /dev/null
+++ b/apps/api/src/api/controllers/admin/apiClientEvents.controller.ts
@@ -0,0 +1,157 @@
+import { Request, Response } from "express";
+import httpStatus from "http-status";
+import { Op, WhereOptions } from "sequelize";
+import logger from "../../../config/logger";
+import ApiClientEvent, { ApiClientEventAttributes } from "../../../models/apiClientEvent.model";
+import { ApiClientErrorType, ApiClientEventStatus, ApiClientOperation } from "../../observability/types";
+
+type ApiClientEventsQuery = {
+  apiKeyPrefix?: string;
+  endDate?: string;
+  errorType?: ApiClientErrorType;
+  limit?: string;
+  offset?: string;
+  operation?: ApiClientOperation;
+  partnerName?: string;
+  quoteId?: string;
+  rampId?: string;
+  requestId?: string;
+  startDate?: string;
+  status?: ApiClientEventStatus;
+};
+
+const DEFAULT_LIMIT = 50;
+const MAX_LIMIT = 200;
+
+const SAFE_EVENT_ATTRIBUTES = [
+  "id",
+  "requestId",
+  "operation",
+  "status",
+  "httpStatus",
+  "errorType",
+  "errorMessage",
+  "partnerName",
+  "apiKeyPrefix",
+  "quoteId",
+  "rampId",
+  "rampType",
+  "network",
+  "paymentMethod",
+  "durationMs",
+  "metadata",
+  "createdAt"
+] satisfies (keyof ApiClientEventAttributes)[];
+
+function parseInteger(value: string | undefined, fallback: number, maximum?: number): number {
+  if (!value) return fallback;
+
+  const parsed = Number.parseInt(value, 10);
+  if (!Number.isFinite(parsed) || parsed < 0) return fallback;
+
+  return maximum ? Math.min(parsed, maximum) : parsed;
+}
+
+function parseDate(value: string | undefined): Date | null {
+  if (!value) return null;
+
+  const date = new Date(value);
+  if (Number.isNaN(date.getTime())) return null;
+
+  return date;
+}
+
+function buildWhere(query: ApiClientEventsQuery): WhereOptions<ApiClientEventAttributes> {
+  const where: WhereOptions<ApiClientEventAttributes> = {};
+
+  if (query.apiKeyPrefix) where.apiKeyPrefix = query.apiKeyPrefix;
+  if (query.errorType) where.errorType = query.errorType;
+  if (query.operation) where.operation = query.operation;
+  if (query.partnerName) where.partnerName = query.partnerName;
+  if (query.quoteId) where.quoteId = query.quoteId;
+  if (query.rampId) where.rampId = query.rampId;
+  if (query.requestId) where.requestId = query.requestId;
+  if (query.status) where.status = query.status;
+
+  const startDate = parseDate(query.startDate);
+  const endDate = parseDate(query.endDate);
+
+  if (startDate || endDate) {
+    where.createdAt = {
+      ...(startDate ? { [Op.gte]: startDate } : {}),
+      ...(endDate ? { [Op.lte]: endDate } : {})
+    };
+  }
+
+  return where;
+}
+
+function buildCounts<T extends string>(
+  events: Pick<ApiClientEventAttributes, "errorType" | "operation" | "status">[],
+  key: string
+) {
+  return events.reduce<Record<T, number>>(
+    (counts, event) => {
+      const value = event[key as keyof typeof event];
+      if (typeof value === "string") {
+        counts[value as T] = (counts[value as T] ?? 0) + 1;
+      }
+      return counts;
+    },
+    {} as Record<T, number>
+  );
+}
+
+/**
+ * GET /v1/admin/api-client-events
+ * List sanitized API client observability events for internal dashboards.
+ */
+export async function listApiClientEvents(
+  req: Request<unknown, unknown, unknown, ApiClientEventsQuery>,
+  res: Response
+): Promise<void> {
+  try {
+    const limit = parseInteger(req.query.limit, DEFAULT_LIMIT, MAX_LIMIT);
+    const offset = parseInteger(req.query.offset, 0);
+    const where = buildWhere(req.query);
+
+    const [eventsResult, summaryEvents] = await Promise.all([
+      ApiClientEvent.findAndCountAll({
+        attributes: SAFE_EVENT_ATTRIBUTES,
+        limit,
+        offset,
+        order: [["createdAt", "DESC"]],
+        where
+      }),
+      ApiClientEvent.findAll({
+        attributes: ["operation", "status", "errorType"],
+        limit: 1000,
+        order: [["createdAt", "DESC"]],
+        where
+      })
+    ]);
+
+    res.status(httpStatus.OK).json({
+      events: eventsResult.rows,
+      limit,
+      offset,
+      summary: {
+        byErrorType: buildCounts<ApiClientErrorType>(summaryEvents, "errorType"),
+        byOperation: buildCounts<ApiClientOperation>(summaryEvents, "operation"),
+        byStatus: buildCounts<ApiClientEventStatus>(summaryEvents, "status"),
+        sampleSize: summaryEvents.length,
+        total: eventsResult.count
+      },
+      total: eventsResult.count
+    });
+  } catch (error) {
+    logger.error("Error listing API client events:", error);
+    res.status(httpStatus.INTERNAL_SERVER_ERROR).json({
+      error: {
+        code: "INTERNAL_SERVER_ERROR",
+        message: "Failed to list API client events",
+        status: httpStatus.INTERNAL_SERVER_ERROR
+      }
+    });
+  }
+}
diff --git a/apps/api/src/api/middlewares/metricsDashboardAuth.ts b/apps/api/src/api/middlewares/metricsDashboardAuth.ts
new file mode 100644
index 000000000..188e33822
--- /dev/null
+++ b/apps/api/src/api/middlewares/metricsDashboardAuth.ts
@@ -0,0 +1,90 @@
+import crypto from "crypto";
+import { NextFunction, Request, Response } from "express";
+import httpStatus from "http-status";
+import logger from "../../config/logger";
+import { config } from "../../config/vars";
+
+/**
+ * Authenticates internal observability dashboard requests with a dedicated bearer token.
+ */
+export function metricsDashboardAuth(req: Request, res: Response, next: NextFunction): void {
+  try {
+    const authHeader = req.headers.authorization;
+
+    if (!authHeader) {
+      logger.warn("Metrics dashboard auth attempt without Authorization header", {
+        ip: req.ip,
+        path: req.path
+      });
+      res.status(httpStatus.UNAUTHORIZED).json({
+        error: {
+          code: "METRICS_DASHBOARD_AUTH_REQUIRED",
+          message: "Metrics dashboard authentication required. Provide Authorization header with Bearer token.",
+          status: httpStatus.UNAUTHORIZED
+        }
+      });
+      return;
+    }
+
+    const parts = authHeader.split(" ");
+    if (parts.length !== 2 || parts[0] !== "Bearer") {
+      res.status(httpStatus.UNAUTHORIZED).json({
+        error: {
+          code: "INVALID_AUTH_FORMAT",
+          message: "Invalid authorization format. Use: Authorization: Bearer <token>",
+          status: httpStatus.UNAUTHORIZED
+        }
+      });
+      return;
+    }
+
+    if (!config.metricsDashboardSecret) {
+      logger.error("METRICS_DASHBOARD_SECRET not configured in environment variables");
+      res.status(httpStatus.INTERNAL_SERVER_ERROR).json({
+        error: {
+          code: "METRICS_DASHBOARD_AUTH_NOT_CONFIGURED",
+          message: "Metrics dashboard authentication is not properly configured",
+          status: httpStatus.INTERNAL_SERVER_ERROR
+        }
+      });
+      return;
+    }
+
+    if (!safeCompare(parts[1], config.metricsDashboardSecret)) {
+      logger.warn("Failed metrics dashboard auth attempt", {
+        ip: req.ip,
+        path: req.path
+      });
+      res.status(httpStatus.FORBIDDEN).json({
+        error: {
+          code: "INVALID_METRICS_DASHBOARD_TOKEN",
+          message: "Invalid metrics dashboard token",
+          status: httpStatus.FORBIDDEN
+        }
+      });
+      return;
+    }
+
+    next();
+  } catch (error) {
+    logger.error("Error in metrics dashboard authentication:", error);
+    res.status(httpStatus.INTERNAL_SERVER_ERROR).json({
+      error: {
+        code: "METRICS_DASHBOARD_AUTH_ERROR",
+        message: "An error occurred during metrics dashboard authentication",
+        status: httpStatus.INTERNAL_SERVER_ERROR
+      }
+    });
+  }
+}
+
+function safeCompare(a: string, b: string): boolean {
+  const bufA = Buffer.from(a);
+  const bufB = Buffer.from(b);
+  if (bufA.length !== bufB.length) {
+    const dummyBuf = Buffer.alloc(bufA.length);
+    crypto.timingSafeEqual(bufA, dummyBuf);
+    return false;
+  }
+  return crypto.timingSafeEqual(bufA, bufB);
+}
diff --git a/apps/api/src/api/routes/v1/admin/api-client-events.route.ts b/apps/api/src/api/routes/v1/admin/api-client-events.route.ts
new file mode 100644
index 000000000..eac5e5784
--- /dev/null
+++ b/apps/api/src/api/routes/v1/admin/api-client-events.route.ts
@@ -0,0 +1,17 @@
+import { Router } from "express";
+import { listApiClientEvents } from "../../../controllers/admin/apiClientEvents.controller";
+import { metricsDashboardAuth } from "../../../middlewares/metricsDashboardAuth";
+
+const router: Router = Router({ mergeParams: true });
+
+router.use(metricsDashboardAuth);
+
+/**
+ * GET /v1/admin/api-client-events
+ * List sanitized API client observability events for internal dashboards.
+ *
+ * Authentication: Requires Authorization: Bearer <METRICS_DASHBOARD_SECRET>
+ */
+router.get("/", listApiClientEvents);
+
+export default router;
diff --git a/apps/api/src/api/routes/v1/index.ts b/apps/api/src/api/routes/v1/index.ts
index b7e6d6e0e..dcfd9a4e4 100644
--- a/apps/api/src/api/routes/v1/index.ts
+++ b/apps/api/src/api/routes/v1/index.ts
@@ -1,6 +1,7 @@
 import { Request, Response, Router } from "express";
 import { sendStatusWithPk as sendMoonbeamStatusWithPk } from "../../controllers/moonbeam.controller";
 import { sendStatusWithPk as sendPendulumStatusWithPk } from "../../controllers/pendulum.controller";
+import apiClientEventsRoutes from "./admin/api-client-events.route";
 import partnerApiKeysRoutes from "./admin/partner-api-keys.route";
 import alfredpayRoutes from "./alfredpay.route";
 import authRoutes from "./auth.route";
@@ -174,6 +175,12 @@ router.use("/metrics", metricsRoutes);
  */
 router.use("/admin/partners/:partnerName/api-keys", partnerApiKeysRoutes);
 
+/**
+ * Admin routes for API client observability dashboards
+ * GET /v1/admin/api-client-events
+ */
+router.use("/admin/api-client-events", apiClientEventsRoutes);
+
 router.get("/ip", (request: Request, response: Response) => {
   response.send(request.ip);
 });
diff --git a/apps/api/src/config/vars.ts b/apps/api/src/config/vars.ts
index 4e254e87c..97495a979 100644
--- a/apps/api/src/config/vars.ts
+++ b/apps/api/src/config/vars.ts
@@ -65,6 +65,7 @@ interface Config {
   rateLimitNumberOfProxies: string | number;
   logs: string;
   adminSecret: string;
+  metricsDashboardSecret: string;
   supabase: {
     url: string;
     anonKey: string;
@@ -156,6 +157,7 @@ export const config: Config = {
     }
   },
   logs: nodeEnv === "production" ? "combined" : "dev",
+  metricsDashboardSecret: process.env.METRICS_DASHBOARD_SECRET || "",
   pendulumWss: process.env.PENDULUM_WSS || "wss://rpc-pendulum.prd.pendulumchain.tech",
   port: process.env.PORT || 3000,
   priceProviders: {
@@ -236,6 +238,7 @@ if (config.env === "production") {
   if (!config.supabase.serviceRoleKey) missing.push("SUPABASE_SERVICE_KEY");
   if (!config.secrets.webhookPrivateKey) missing.push("WEBHOOK_PRIVATE_KEY");
   if (!config.adminSecret) missing.push("ADMIN_SECRET");
+  if (!config.metricsDashboardSecret) missing.push("METRICS_DASHBOARD_SECRET");
   if (!process.env.FLOW_VARIANT) missing.push("FLOW_VARIANT");
 
   if (missing.length > 0) {

From 73bfb593497179eaa2a2afedb25e1d9dd470428b Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 18:21:34 +0200
Subject: [PATCH 06/11] Add metrics dashboard endpoint tests

---
 .../admin/apiClientEvents.controller.test.ts  | 187 ++++++++++++++++++
 apps/api/src/config/vars.test.ts              |  12 ++
 2 files changed, 199 insertions(+)
 create mode 100644 apps/api/src/api/controllers/admin/apiClientEvents.controller.test.ts

diff --git a/apps/api/src/api/controllers/admin/apiClientEvents.controller.test.ts b/apps/api/src/api/controllers/admin/apiClientEvents.controller.test.ts
new file mode 100644
index 000000000..e808705f7
--- /dev/null
+++ b/apps/api/src/api/controllers/admin/apiClientEvents.controller.test.ts
@@ -0,0 +1,187 @@
+import { afterEach, describe, expect, it, mock } from "bun:test";
+import express from "express";
+import httpStatus from "http-status";
+import { config } from "../../../config/vars";
+import ApiClientEvent from "../../../models/apiClientEvent.model";
+import apiClientEventsRoutes from "../../routes/v1/admin/api-client-events.route";
+import { listApiClientEvents } from "./apiClientEvents.controller";
+
+type ResponseRecorder = {
+  body: unknown;
+  statusCode: number;
+  json: ReturnType<typeof mock>;
+  status: ReturnType<typeof mock>;
+};
+
+function createResponse() {
+  const res: ResponseRecorder = {
+    body: undefined,
+    json: mock((body: unknown) => {
+      res.body = body;
+      return res;
+    }),
+    status: mock((statusCode: number) => {
+      res.statusCode = statusCode;
+      return res;
+    }),
+    statusCode: Number(httpStatus.OK)
+  };
+
+  return res;
+}
+
+describe("api client events admin route", () => {
+  const originalAdminSecret = config.adminSecret;
+  const originalMetricsDashboardSecret = config.metricsDashboardSecret;
+
+  afterEach(() => {
+    config.adminSecret = originalAdminSecret;
+    config.metricsDashboardSecret = originalMetricsDashboardSecret;
+  });
+
+  async function fetchFromTestRoute(authorization?: string) {
+    const app = express();
+    app.use("/v1/admin/api-client-events", apiClientEventsRoutes);
+
+    const server = app.listen(0);
+    const address = server.address();
+
+    if (!address || typeof address === "string") {
+      server.close();
+      throw new Error("Could not bind test server");
+    }
+
+    try {
+      return await fetch(`http://127.0.0.1:${address.port}/v1/admin/api-client-events`, {
+        ...(authorization ? { headers: { Authorization: authorization } } : {})
+      });
+    } finally {
+      server.close();
+    }
+  }
+
+  it("rejects requests without metrics dashboard authentication before querying events", async () => {
+    const response = await fetchFromTestRoute();
+    const body = await response.json();
+
+    expect(response.status).toBe(httpStatus.UNAUTHORIZED);
+    expect(body.error.code).toBe("METRICS_DASHBOARD_AUTH_REQUIRED");
+  });
+
+  it("rejects the admin secret because metrics access has a dedicated secret", async () => {
+    config.adminSecret = "admin-secret";
+    config.metricsDashboardSecret = "metrics-dashboard-secret";
+
+    const response = await fetchFromTestRoute("Bearer admin-secret");
+    const body = await response.json();
+
+    expect(response.status).toBe(httpStatus.FORBIDDEN);
+    expect(body.error.code).toBe("INVALID_METRICS_DASHBOARD_TOKEN");
+  });
+});
+
+describe("listApiClientEvents", () => {
+  const originalFindAndCountAll = ApiClientEvent.findAndCountAll;
+  const originalFindAll = ApiClientEvent.findAll;
+
+  afterEach(() => {
+    ApiClientEvent.findAndCountAll = originalFindAndCountAll;
+    ApiClientEvent.findAll = originalFindAll;
+  });
+
+  it("returns safe event fields, pagination, filters, and summary counts", async () => {
+    const findAndCountAllMock = mock(async (_options: unknown) => ({
+      count: 2,
+      rows: [
+        {
+          apiKeyPrefix: "sk_live_1234",
+          createdAt: new Date("2026-01-02T00:00:00.000Z"),
+          durationMs: 25,
+          errorMessage: null,
+          errorType: "none",
+          httpStatus: 200,
+          id: "event-1",
+          metadata: { endpoint: "/v1/quotes" },
+          operation: "quote_create",
+          partnerName: "Partner",
+          quoteId: "quote-1",
+          rampId: null,
+          requestId: "request-1",
+          status: "success"
+        }
+      ]
+    }));
+    const findAllMock = mock(async (_options: unknown) => [
+      { errorType: "none", operation: "quote_create", status: "success" },
+      { errorType: "internal_error", operation: "ramp_start", status: "failure" }
+    ]);
+
+    ApiClientEvent.findAndCountAll = findAndCountAllMock as unknown as typeof ApiClientEvent.findAndCountAll;
+    ApiClientEvent.findAll = findAllMock as unknown as typeof ApiClientEvent.findAll;
+
+    const res = createResponse();
+    await listApiClientEvents(
+      {
+        query: {
+          endDate: "2026-01-03T00:00:00.000Z",
+          limit: "500",
+          offset: "5",
+          operation: "quote_create",
+          partnerName: "Partner",
+          startDate: "2026-01-01T00:00:00.000Z",
+          status: "success"
+        }
+      } as Parameters<typeof listApiClientEvents>[0],
+      res as unknown as Parameters<typeof listApiClientEvents>[1]
+    );
+
+    const firstFindCall = findAndCountAllMock.mock.calls[0];
+    expect(firstFindCall).toBeDefined();
+
+    const findOptions = firstFindCall?.[0] as {
+      attributes: string[];
+      limit: number;
+      offset: number;
+      where: Record<string, unknown>;
+    };
+
+    expect(res.statusCode).toBe(httpStatus.OK);
+    expect(findOptions.limit).toBe(200);
+    expect(findOptions.offset).toBe(5);
+    expect(findOptions.attributes).not.toContain("partnerId");
+    expect(findOptions.attributes).not.toContain("userId");
+    expect(findOptions.where.operation).toBe("quote_create");
+    expect(findOptions.where.partnerName).toBe("Partner");
+    expect(findOptions.where.status).toBe("success");
+    expect(res.body).toEqual({
+      events: [
+        {
+          apiKeyPrefix: "sk_live_1234",
+          createdAt: new Date("2026-01-02T00:00:00.000Z"),
+          durationMs: 25,
+          errorMessage: null,
+          errorType: "none",
+          httpStatus: 200,
+          id: "event-1",
+          metadata: { endpoint: "/v1/quotes" },
+          operation: "quote_create",
+          partnerName: "Partner",
+          quoteId: "quote-1",
+          rampId: null,
+          requestId: "request-1",
+          status: "success"
+        }
+      ],
+      limit: 200,
+      offset: 5,
+      summary: {
+        byErrorType: { internal_error: 1, none: 1 },
+        byOperation: { quote_create: 1, ramp_start: 1 },
+        byStatus: { failure: 1, success: 1 },
+        sampleSize: 2,
+        total: 2
+      },
+      total: 2
+    });
+  });
+});
diff --git a/apps/api/src/config/vars.test.ts b/apps/api/src/config/vars.test.ts
index 41ed0cc5f..0a3254e6d 100644
--- a/apps/api/src/config/vars.test.ts
+++ b/apps/api/src/config/vars.test.ts
@@ -5,6 +5,7 @@ const bunExecutable = Bun.argv[0];
 
 const requiredProductionEnv = {
   ADMIN_SECRET: "test-admin-secret",
+  METRICS_DASHBOARD_SECRET: "test-metrics-dashboard-secret",
   SUPABASE_ANON_KEY: "test-anon-key",
   SUPABASE_SERVICE_KEY: "test-service-key",
   SUPABASE_URL: "https://example.supabase.co",
@@ -67,4 +68,15 @@ describe("vars deployment environment validation", () => {
     expect(result.exitCode).toBe(1);
     expect(result.stderr).toContain("DEPLOYMENT_ENV=sandbox requires SANDBOX_ENABLED=true");
   });
+
+  it("requires the metrics dashboard secret in production", async () => {
+    const result = await importVarsWithEnv({
+      DEPLOYMENT_ENV: "production",
+      METRICS_DASHBOARD_SECRET: "",
+      NODE_ENV: "production"
+    });
+
+    expect(result.exitCode).toBe(1);
+    expect(result.stderr).toContain("METRICS_DASHBOARD_SECRET");
+  });
 });

From 6482764a4987ee90dafbaaf29a064e510021f731 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 18:21:43 +0200
Subject: [PATCH 07/11] Add internal metrics dashboard app

---
 apps/dashboard/.env.example                   |   2 +
 apps/dashboard/App.css                        |  47 +++
 apps/dashboard/_redirects                     |   9 +
 apps/dashboard/index.html                     |  13 +
 apps/dashboard/package.json                   |  29 ++
 apps/dashboard/public/404.html                |  25 ++
 apps/dashboard/src/App.tsx                    | 328 ++++++++++++++++++
 .../dashboard/src/api/apiClientEvents.test.ts |  43 +++
 apps/dashboard/src/api/apiClientEvents.ts     |  83 +++++
 apps/dashboard/src/config/index.ts            |  19 +
 apps/dashboard/src/helpers/cn.ts              |   6 +
 apps/dashboard/src/main.tsx                   |  27 ++
 apps/dashboard/tsconfig.json                  |  27 ++
 apps/dashboard/tsconfig.node.json             |   9 +
 apps/dashboard/vite.config.ts                 |  19 +
 bun.lock                                      |  23 ++
 package.json                                  |   7 +-
 17 files changed, 714 insertions(+), 2 deletions(-)
 create mode 100644 apps/dashboard/.env.example
 create mode 100644 apps/dashboard/App.css
 create mode 100644 apps/dashboard/_redirects
 create mode 100644 apps/dashboard/index.html
 create mode 100644 apps/dashboard/package.json
 create mode 100644 apps/dashboard/public/404.html
 create mode 100644 apps/dashboard/src/App.tsx
 create mode 100644 apps/dashboard/src/api/apiClientEvents.test.ts
 create mode 100644 apps/dashboard/src/api/apiClientEvents.ts
 create mode 100644 apps/dashboard/src/config/index.ts
 create mode 100644 apps/dashboard/src/helpers/cn.ts
 create mode 100644 apps/dashboard/src/main.tsx
 create mode 100644 apps/dashboard/tsconfig.json
 create mode 100644 apps/dashboard/tsconfig.node.json
 create mode 100644 apps/dashboard/vite.config.ts

diff --git a/apps/dashboard/.env.example b/apps/dashboard/.env.example
new file mode 100644
index 000000000..6f77fe506
--- /dev/null
+++ b/apps/dashboard/.env.example
@@ -0,0 +1,2 @@
+VITE_DASHBOARD_API_BASE=http://localhost:3000
+VITE_ENVIRONMENT=development
diff --git a/apps/dashboard/App.css b/apps/dashboard/App.css
new file mode 100644
index 000000000..30e018e16
--- /dev/null
+++ b/apps/dashboard/App.css
@@ -0,0 +1,47 @@
+@import "tailwindcss";
+
+:root {
+  color: #e7f7ee;
+  background: #06100d;
+  font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+  font-synthesis: none;
+  text-rendering: optimizeLegibility;
+  -webkit-font-smoothing: antialiased;
+  -moz-osx-font-smoothing: grayscale;
+}
+
+body {
+  margin: 0;
+  min-width: 320px;
+  min-height: 100vh;
+}
+
+button,
+input,
+select {
+  font: inherit;
+}
+
+button {
+  transition:
+    background-color 120ms ease,
+    border-color 120ms ease,
+    color 120ms ease,
+    opacity 120ms ease,
+    transform 80ms ease,
+    box-shadow 120ms ease;
+}
+
+button:not(:disabled):active {
+  transform: translateY(1px);
+}
+
+#app {
+  min-height: 100vh;
+}
+
+.dashboard-shell {
+  min-height: 100vh;
+  background: radial-gradient(circle at top left, rgba(36, 201, 137, 0.2), transparent 32rem),
+    linear-gradient(135deg, #06100d 0%, #0c1714 48%, #09110f 100%);
+}
diff --git a/apps/dashboard/_redirects b/apps/dashboard/_redirects
new file mode 100644
index 000000000..ade7e4cba
--- /dev/null
+++ b/apps/dashboard/_redirects
@@ -0,0 +1,9 @@
+/api/production/*  https://api.vortexfinance.co/:splat  200
+/api/staging/*     https://api-staging.vortexfinance.co/:splat  200
+/api/sandbox/*     https://api-sandbox.vortexfinance.co/:splat  200
+
+# Known dashboard routes — serve SPA shell with real 200
+/                              /index.html  200
+
+# Everything else is a real 404
+/*                             /404.html    404
diff --git a/apps/dashboard/index.html b/apps/dashboard/index.html
new file mode 100644
index 000000000..1d0cf4ce7
--- /dev/null
+++ b/apps/dashboard/index.html
@@ -0,0 +1,13 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="robots" content="noindex,nofollow" />
+    <title>Vortex Internal Dashboard</title>
+  </head>
+  <body>
+    <div id="app"></div>
+    <script type="module" src="/src/main.tsx"></script>
+  </body>
+</html>
diff --git a/apps/dashboard/package.json b/apps/dashboard/package.json
new file mode 100644
index 000000000..f10ef5a31
--- /dev/null
+++ b/apps/dashboard/package.json
@@ -0,0 +1,29 @@
+{
+  "dependencies": {
+    "@tailwindcss/vite": "^4.0.3",
+    "@tanstack/react-query": "^5.64.2",
+    "@vitejs/plugin-react": "^4.3.4",
+    "clsx": "^2.1.1",
+    "react": "=19.2.0",
+    "react-dom": "=19.2.0",
+    "tailwind-merge": "^3.4.0",
+    "tailwindcss": "^4.0.3"
+  },
+  "devDependencies": {
+    "@types/node": "catalog:",
+    "@types/react": "^19.0.8",
+    "@types/react-dom": "^19.0.3",
+    "typescript": "catalog:",
+    "vite": "^6.2.6"
+  },
+  "name": "vortex-dashboard",
+  "private": true,
+  "scripts": {
+    "build": "bun x --bun vite build && cp _redirects dist/_redirects",
+    "dev": "bun x --bun vite --host",
+    "preview": "bun x --bun vite preview",
+    "test": "bun test"
+  },
+  "type": "module",
+  "version": "1.0.0"
+}
diff --git a/apps/dashboard/public/404.html b/apps/dashboard/public/404.html
new file mode 100644
index 000000000..00bb59dc9
--- /dev/null
+++ b/apps/dashboard/public/404.html
@@ -0,0 +1,25 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Not found</title>
+    <style>
+      body {
+        align-items: center;
+        background: #07100d;
+        color: #e7f7ee;
+        display: flex;
+        font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
+        justify-content: center;
+        min-height: 100vh;
+      }
+    </style>
+  </head>
+  <body>
+    <main>
+      <h1>404</h1>
+      <p>This internal dashboard route does not exist.</p>
+    </main>
+  </body>
+</html>
diff --git a/apps/dashboard/src/App.tsx b/apps/dashboard/src/App.tsx
new file mode 100644
index 000000000..819dba10d
--- /dev/null
+++ b/apps/dashboard/src/App.tsx
@@ -0,0 +1,328 @@
+import { useQuery, useQueryClient } from "@tanstack/react-query";
+import { type FormEvent, type ReactNode, useMemo, useState } from "react";
+import { ApiClientEventsFilters, fetchApiClientEvents } from "./api/apiClientEvents";
+import { cn } from "./helpers/cn";
+
+const tokenStorageKey = "vortex-dashboard-metrics-token";
+
+const defaultFilters: ApiClientEventsFilters = {
+  limit: 50,
+  offset: 0
+};
+
+const operations = [
+  "auth_api_key",
+  "auth_public_key",
+  "auth_dual",
+  "auth_ownership",
+  "quote_create",
+  "quote_create_best",
+  "quote_get",
+  "ramp_register",
+  "ramp_update",
+  "ramp_start",
+  "ramp_status",
+  "ramp_errors"
+];
+
+function readStoredToken(): string {
+  return window.sessionStorage.getItem(tokenStorageKey) ?? "";
+}
+
+export function App() {
+  const queryClient = useQueryClient();
+  const [metricsToken, setMetricsToken] = useState(readStoredToken);
+  const [tokenDraft, setTokenDraft] = useState(metricsToken);
+  const [filters, setFilters] = useState<ApiClientEventsFilters>(defaultFilters);
+
+  const eventsQuery = useQuery({
+    enabled: metricsToken.length > 0,
+    queryFn: () => fetchApiClientEvents(filters, metricsToken),
+    queryKey: ["api-client-events", filters, metricsToken]
+  });
+
+  const failureRate = useMemo(() => {
+    const summary = eventsQuery.data?.summary;
+    if (!summary || summary.sampleSize === 0) return "0.0%";
+
+    const failures = summary.byStatus.failure ?? 0;
+    return `${((failures / summary.sampleSize) * 100).toFixed(1)}%`;
+  }, [eventsQuery.data]);
+
+  function handleTokenSubmit(event: FormEvent<HTMLFormElement>) {
+    event.preventDefault();
+    const trimmedToken = tokenDraft.trim();
+    window.sessionStorage.setItem(tokenStorageKey, trimmedToken);
+    setMetricsToken(trimmedToken);
+  }
+
+  function handleFiltersSubmit(event: FormEvent<HTMLFormElement>) {
+    event.preventDefault();
+    const formData = new FormData(event.currentTarget);
+    setFilters({
+      apiKeyPrefix: stringValue(formData, "apiKeyPrefix"),
+      endDate: stringValue(formData, "endDate"),
+      errorType: stringValue(formData, "errorType"),
+      limit: numberValue(formData, "limit", 50),
+      offset: 0,
+      operation: stringValue(formData, "operation"),
+      partnerName: stringValue(formData, "partnerName"),
+      quoteId: stringValue(formData, "quoteId"),
+      rampId: stringValue(formData, "rampId"),
+      requestId: stringValue(formData, "requestId"),
+      startDate: stringValue(formData, "startDate"),
+      status: stringValue(formData, "status")
+    });
+  }
+
+  function handleClearToken() {
+    window.sessionStorage.removeItem(tokenStorageKey);
+    setMetricsToken("");
+    setTokenDraft("");
+  }
+
+  function movePage(direction: -1 | 1) {
+    setFilters(currentFilters => ({
+      ...currentFilters,
+      offset: Math.max(0, currentFilters.offset + direction * currentFilters.limit)
+    }));
+  }
+
+  return (
+    <main className="dashboard-shell px-6 py-8 text-emerald-50 lg:px-10">
+      <section className="mx-auto flex max-w-7xl flex-col gap-8">
+        <header className="flex flex-col gap-4 border-emerald-400/20 border-b pb-6 md:flex-row md:items-end md:justify-between">
+          <div>
+            <p className="mb-2 font-mono text-emerald-300 text-xs uppercase tracking-[0.35em]">Internal observability</p>
+            <h1 className="font-semibold text-4xl tracking-tight">API client events</h1>
+            <p className="mt-3 max-w-2xl text-emerald-100/70">
+              Sanitized partner API activity, protected by a dedicated metrics dashboard bearer token. Tokens are kept in
+              session storage only.
+            </p>
+          </div>
+          <form className="flex w-full flex-col gap-3 md:w-[28rem]" onSubmit={handleTokenSubmit}>
+            <label className="font-mono text-emerald-200/80 text-xs uppercase tracking-[0.24em]" htmlFor="metrics-token">
+              Metrics dashboard token
+            </label>
+            <div className="flex gap-2">
+              <input
+                className="min-w-0 flex-1 rounded-lg border border-emerald-300/20 bg-black/30 px-3 py-2 text-sm outline-none ring-emerald-300/40 transition focus:ring-2"
+                id="metrics-token"
+                onChange={event => setTokenDraft(event.target.value)}
+                placeholder="Bearer token value"
+                type="password"
+                value={tokenDraft}
+              />
+              <button
+                className="rounded-lg bg-emerald-300 px-4 py-2 font-medium text-emerald-950 text-sm shadow-emerald-300/10 shadow-sm hover:bg-emerald-200 hover:shadow-emerald-300/20 hover:shadow-md active:bg-emerald-100"
+                type="submit"
+              >
+                Unlock
+              </button>
+              <button
+                className="rounded-lg border border-emerald-300/20 px-3 py-2 text-emerald-100/80 text-sm hover:border-emerald-300/40 hover:bg-emerald-300/10 hover:text-emerald-50 active:bg-emerald-300/15"
+                onClick={handleClearToken}
+                type="button"
+              >
+                Clear
+              </button>
+            </div>
+          </form>
+        </header>
+
+        <form
+          className="grid gap-3 rounded-2xl border border-emerald-400/15 bg-black/20 p-4 md:grid-cols-4"
+          onSubmit={handleFiltersSubmit}
+        >
+          <FilterInput name="partnerName" placeholder="Partner" />
+          <FilterSelect name="status" options={["success", "failure"]} placeholder="Any status" />
+          <FilterSelect name="operation" options={operations} placeholder="Any operation" />
+          <FilterInput name="errorType" placeholder="Error type" />
+          <FilterInput name="apiKeyPrefix" placeholder="API key prefix" />
+          <FilterInput name="requestId" placeholder="Request ID" />
+          <FilterInput name="quoteId" placeholder="Quote ID" />
+          <FilterInput name="rampId" placeholder="Ramp ID" />
+          <FilterInput name="startDate" type="datetime-local" />
+          <FilterInput name="endDate" type="datetime-local" />
+          <FilterSelect name="limit" options={["25", "50", "100", "200"]} placeholder="50 rows" />
+          <button
+            className="rounded-lg bg-emerald-300 px-4 py-2 font-semibold text-emerald-950 text-sm shadow-emerald-300/10 shadow-sm hover:bg-emerald-200 hover:shadow-emerald-300/20 hover:shadow-md active:bg-emerald-100"
+            type="submit"
+          >
+            Apply filters
+          </button>
+        </form>
+
+        <section className="grid gap-4 md:grid-cols-4">
+          <MetricCard label="Total matching rows" value={eventsQuery.data?.summary.total ?? 0} />
+          <MetricCard label="Summary sample" value={eventsQuery.data?.summary.sampleSize ?? 0} />
+          <MetricCard label="Failures in sample" value={eventsQuery.data?.summary.byStatus.failure ?? 0} />
+          <MetricCard label="Failure rate" value={failureRate} />
+        </section>
+
+        <section className="overflow-hidden rounded-2xl border border-emerald-400/15 bg-black/25">
+          <div className="flex flex-col gap-3 border-emerald-400/15 border-b p-4 md:flex-row md:items-center md:justify-between">
+            <div>
+              <h2 className="font-semibold text-lg">Recent events</h2>
+              <p className="text-emerald-100/60 text-sm">
+                Showing {eventsQuery.data?.events.length ?? 0} rows from offset {filters.offset}.
+              </p>
+            </div>
+            <div className="flex gap-2">
+              <button
+                className="rounded-lg border border-emerald-300/20 px-3 py-2 text-sm hover:border-emerald-300/40 hover:bg-emerald-300/10 active:bg-emerald-300/15 disabled:opacity-40"
+                disabled={filters.offset === 0}
+                onClick={() => movePage(-1)}
+                type="button"
+              >
+                Previous
+              </button>
+              <button
+                className="rounded-lg border border-emerald-300/20 px-3 py-2 text-sm hover:border-emerald-300/40 hover:bg-emerald-300/10 active:bg-emerald-300/15 disabled:opacity-40"
+                disabled={!eventsQuery.data || filters.offset + filters.limit >= eventsQuery.data.total}
+                onClick={() => movePage(1)}
+                type="button"
+              >
+                Next
+              </button>
+              <button
+                className="rounded-lg bg-emerald-300/10 px-3 py-2 text-emerald-100 text-sm hover:bg-emerald-300/20 active:bg-emerald-300/25"
+                onClick={() => queryClient.invalidateQueries({ queryKey: ["api-client-events"] })}
+                type="button"
+              >
+                Refresh
+              </button>
+            </div>
+          </div>
+
+          {eventsQuery.isError ? <ErrorState message={eventsQuery.error.message} /> : null}
+          {eventsQuery.isLoading ? <EmptyState message="Loading event data…" /> : null}
+          {!metricsToken ? <EmptyState message="Enter the metrics dashboard token to load protected dashboard data." /> : null}
+          {eventsQuery.data ? <EventsTable events={eventsQuery.data.events} /> : null}
+        </section>
+      </section>
+    </main>
+  );
+}
+
+function stringValue(formData: FormData, key: string): string | undefined {
+  const value = formData.get(key);
+  if (typeof value !== "string") return undefined;
+
+  const trimmedValue = value.trim();
+  return trimmedValue.length > 0 ? trimmedValue : undefined;
+}
+
+function numberValue(formData: FormData, key: string, fallback: number): number {
+  const value = stringValue(formData, key);
+  if (!value) return fallback;
+
+  const parsed = Number.parseInt(value, 10);
+  return Number.isFinite(parsed) ? parsed : fallback;
+}
+
+function FilterInput({ name, placeholder, type = "text" }: { name: string; placeholder?: string; type?: string }) {
+  return (
+    <input
+      className="rounded-lg border border-emerald-300/15 bg-black/30 px-3 py-2 text-sm outline-none ring-emerald-300/40 transition placeholder:text-emerald-100/35 focus:ring-2"
+      name={name}
+      placeholder={placeholder}
+      type={type}
+    />
+  );
+}
+
+function FilterSelect({ name, options, placeholder }: { name: string; options: string[]; placeholder: string }) {
+  return (
+    <select
+      className="rounded-lg border border-emerald-300/15 bg-black/30 px-3 py-2 text-sm outline-none ring-emerald-300/40 transition focus:ring-2"
+      name={name}
+    >
+      <option value="">{placeholder}</option>
+      {options.map(option => (
+        <option key={option} value={option}>
+          {option}
+        </option>
+      ))}
+    </select>
+  );
+}
+
+function MetricCard({ label, value }: { label: string; value: number | string }) {
+  return (
+    <div className="rounded-2xl border border-emerald-400/15 bg-black/25 p-5">
+      <p className="font-mono text-emerald-200/60 text-xs uppercase tracking-[0.24em]">{label}</p>
+      <p className="mt-3 font-semibold text-3xl">{value}</p>
+    </div>
+  );
+}
+
+function EventsTable({ events }: { events: Awaited<ReturnType<typeof fetchApiClientEvents>>["events"] }) {
+  if (events.length === 0) {
+    return <EmptyState message="No events matched the selected filters." />;
+  }
+
+  return (
+    <div className="overflow-x-auto">
+      <table className="min-w-full text-left text-sm">
+        <thead className="bg-emerald-950/40 text-emerald-100/60">
+          <tr>
+            <HeaderCell>Created</HeaderCell>
+            <HeaderCell>Status</HeaderCell>
+            <HeaderCell>Operation</HeaderCell>
+            <HeaderCell>Partner</HeaderCell>
+            <HeaderCell>HTTP</HeaderCell>
+            <HeaderCell>Error</HeaderCell>
+            <HeaderCell>Duration</HeaderCell>
+            <HeaderCell>Ramp / Quote</HeaderCell>
+          </tr>
+        </thead>
+        <tbody>
+          {events.map(event => (
+            <tr className="border-emerald-400/10 border-t" key={event.id}>
+              <BodyCell>{new Date(event.createdAt).toLocaleString()}</BodyCell>
+              <BodyCell>
+                <span
+                  className={cn(
+                    "rounded-full px-2 py-1 font-medium text-xs",
+                    event.status === "success" ? "bg-emerald-300/15 text-emerald-200" : "bg-red-400/15 text-red-200"
+                  )}
+                >
+                  {event.status}
+                </span>
+              </BodyCell>
+              <BodyCell>{event.operation}</BodyCell>
+              <BodyCell>{event.partnerName ?? "—"}</BodyCell>
+              <BodyCell>{event.httpStatus ?? "—"}</BodyCell>
+              <BodyCell>
+                <div>{event.errorType ?? "—"}</div>
+                {event.errorMessage ? <div className="mt-1 max-w-xs text-emerald-100/45">{event.errorMessage}</div> : null}
+              </BodyCell>
+              <BodyCell>{event.durationMs ? `${event.durationMs}ms` : "—"}</BodyCell>
+              <BodyCell>
+                <div>{event.rampId ?? "—"}</div>
+                <div className="mt-1 text-emerald-100/45">{event.quoteId ?? "—"}</div>
+              </BodyCell>
+            </tr>
+          ))}
+        </tbody>
+      </table>
+    </div>
+  );
+}
+
+function HeaderCell({ children }: { children: ReactNode }) {
+  return <th className="px-4 py-3 font-mono font-normal text-xs uppercase tracking-[0.18em]">{children}</th>;
+}
+
+function BodyCell({ children }: { children: ReactNode }) {
+  return <td className="px-4 py-3 align-top text-emerald-50/85">{children}</td>;
+}
+
+function EmptyState({ message }: { message: string }) {
+  return <div className="p-8 text-center text-emerald-100/60">{message}</div>;
+}
+
+function ErrorState({ message }: { message: string }) {
+  return <div className="border-red-300/20 border-b bg-red-950/30 p-4 text-red-100 text-sm">{message}</div>;
+}
diff --git a/apps/dashboard/src/api/apiClientEvents.test.ts b/apps/dashboard/src/api/apiClientEvents.test.ts
new file mode 100644
index 000000000..db89d6817
--- /dev/null
+++ b/apps/dashboard/src/api/apiClientEvents.test.ts
@@ -0,0 +1,43 @@
+import { afterEach, describe, expect, it, mock } from "bun:test";
+import { buildApiClientEventsUrl, fetchApiClientEvents } from "./apiClientEvents";
+
+describe("buildApiClientEventsUrl", () => {
+  it("builds relative Netlify proxy URLs with query filters", () => {
+    const url = buildApiClientEventsUrl({ limit: 50, offset: 0, partnerName: "Partner" }, "/api/staging");
+
+    expect(url).toBe("/api/staging/v1/admin/api-client-events?limit=50&offset=0&partnerName=Partner");
+  });
+});
+
+describe("fetchApiClientEvents", () => {
+  const originalFetch = globalThis.fetch;
+
+  afterEach(() => {
+    globalThis.fetch = originalFetch;
+  });
+
+  it("sends the metrics dashboard bearer token to the protected endpoint", async () => {
+    const fetchMock = mock(async (_input: string | URL | Request, _init?: RequestInit) =>
+      Response.json({
+        events: [],
+        limit: 50,
+        offset: 0,
+        summary: { byErrorType: {}, byOperation: {}, byStatus: {}, sampleSize: 0, total: 0 },
+        total: 0
+      })
+    );
+    globalThis.fetch = fetchMock as unknown as typeof fetch;
+
+    await fetchApiClientEvents({ limit: 50, offset: 0 }, "metrics-dashboard-secret");
+
+    const firstFetchCall = fetchMock.mock.calls[0];
+    expect(firstFetchCall).toBeDefined();
+
+    const init = firstFetchCall?.[1];
+    expect(init).toEqual({
+      headers: {
+        Authorization: "Bearer metrics-dashboard-secret"
+      }
+    });
+  });
+});
diff --git a/apps/dashboard/src/api/apiClientEvents.ts b/apps/dashboard/src/api/apiClientEvents.ts
new file mode 100644
index 000000000..39ceb9870
--- /dev/null
+++ b/apps/dashboard/src/api/apiClientEvents.ts
@@ -0,0 +1,83 @@
+import { config } from "../config";
+
+export type ApiClientEvent = {
+  apiKeyPrefix: string | null;
+  createdAt: string;
+  durationMs: number | null;
+  errorMessage: string | null;
+  errorType: string | null;
+  httpStatus: number | null;
+  id: string;
+  metadata: Record<string, unknown> | null;
+  network: string | null;
+  operation: string;
+  partnerName: string | null;
+  paymentMethod: string | null;
+  quoteId: string | null;
+  rampId: string | null;
+  rampType: string | null;
+  requestId: string | null;
+  status: "success" | "failure";
+};
+
+export type ApiClientEventsFilters = {
+  apiKeyPrefix?: string;
+  endDate?: string;
+  errorType?: string;
+  limit: number;
+  offset: number;
+  operation?: string;
+  partnerName?: string;
+  quoteId?: string;
+  rampId?: string;
+  requestId?: string;
+  startDate?: string;
+  status?: string;
+};
+
+export type ApiClientEventsResponse = {
+  events: ApiClientEvent[];
+  limit: number;
+  offset: number;
+  summary: {
+    byErrorType: Record<string, number>;
+    byOperation: Record<string, number>;
+    byStatus: Record<string, number>;
+    sampleSize: number;
+    total: number;
+  };
+  total: number;
+};
+
+export function buildApiClientEventsUrl(filters: ApiClientEventsFilters, apiBaseUrl = config.apiBaseUrl): string {
+  const params = new URLSearchParams();
+
+  for (const [key, value] of Object.entries(filters)) {
+    if (value !== undefined && value !== "") {
+      params.set(key, String(value));
+    }
+  }
+
+  const queryString = params.toString();
+  const baseUrl = apiBaseUrl.replace(/\/$/, "");
+  const url = `${baseUrl}/v1/admin/api-client-events`;
+  return queryString ? `${url}?${queryString}` : url;
+}
+
+export async function fetchApiClientEvents(
+  filters: ApiClientEventsFilters,
+  metricsDashboardToken: string
+): Promise<ApiClientEventsResponse> {
+  const response = await fetch(buildApiClientEventsUrl(filters), {
+    headers: {
+      Authorization: `Bearer ${metricsDashboardToken}`
+    }
+  });
+
+  if (!response.ok) {
+    const body = await response.text();
+    throw new Error(body || `Request failed with status ${response.status}`);
+  }
+
+  return response.json();
+}
diff --git a/apps/dashboard/src/config/index.ts b/apps/dashboard/src/config/index.ts
new file mode 100644
index 000000000..88b0ef3bc
--- /dev/null
+++ b/apps/dashboard/src/config/index.ts
@@ -0,0 +1,19 @@
+type Environment = "development" | "staging" | "production" | "sandbox";
+
+const nodeEnv = process.env.NODE_ENV as Environment;
+const env = (import.meta.env.VITE_ENVIRONMENT || nodeEnv || "development") as Environment;
+const explicitApiBase = import.meta.env.VITE_DASHBOARD_API_BASE;
+
+function getDefaultApiBase(environment: Environment): string {
+  if (environment === "production") return "/api/production";
+  if (environment === "staging") return "/api/staging";
+  if (environment === "sandbox") return "/api/sandbox";
+  return "http://localhost:3000";
+}
+
+export const config = {
+  apiBaseUrl: explicitApiBase || getDefaultApiBase(env),
+  env,
+  isDev: env === "development",
+  isProd: env === "production"
+};
diff --git a/apps/dashboard/src/helpers/cn.ts b/apps/dashboard/src/helpers/cn.ts
new file mode 100644
index 000000000..365058ceb
--- /dev/null
+++ b/apps/dashboard/src/helpers/cn.ts
@@ -0,0 +1,6 @@
+import { type ClassValue, clsx } from "clsx";
+import { twMerge } from "tailwind-merge";
+
+export function cn(...inputs: ClassValue[]) {
+  return twMerge(clsx(inputs));
+}
diff --git a/apps/dashboard/src/main.tsx b/apps/dashboard/src/main.tsx
new file mode 100644
index 000000000..78bb9715b
--- /dev/null
+++ b/apps/dashboard/src/main.tsx
@@ -0,0 +1,27 @@
+import "../App.css";
+
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { createRoot } from "react-dom/client";
+import { App } from "./App";
+
+const queryClient = new QueryClient({
+  defaultOptions: {
+    queries: {
+      refetchOnWindowFocus: false,
+      retry: 1,
+      staleTime: 30_000
+    }
+  }
+});
+
+const root = document.getElementById("app");
+
+if (!root) {
+  throw new Error("Root element not found");
+}
+
+createRoot(root).render(
+  <QueryClientProvider client={queryClient}>
+    <App />
+  </QueryClientProvider>
+);
diff --git a/apps/dashboard/tsconfig.json b/apps/dashboard/tsconfig.json
new file mode 100644
index 000000000..8347c8c73
--- /dev/null
+++ b/apps/dashboard/tsconfig.json
@@ -0,0 +1,27 @@
+{
+  "compilerOptions": {
+    "allowJs": false,
+    "allowSyntheticDefaultImports": true,
+    "esModuleInterop": false,
+    "forceConsistentCasingInFileNames": true,
+    "isolatedModules": true,
+    "jsx": "react-jsx",
+    "lib": ["DOM", "DOM.Iterable", "ESNext"],
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "noEmit": true,
+    "resolveJsonModule": true,
+    "skipLibCheck": true,
+    "strict": true,
+    "target": "ESNext",
+    "types": ["node", "bun"],
+    "useDefineForClassFields": true
+  },
+  "exclude": ["node_modules"],
+  "include": ["src"],
+  "references": [
+    {
+      "path": "./tsconfig.node.json"
+    }
+  ]
+}
diff --git a/apps/dashboard/tsconfig.node.json b/apps/dashboard/tsconfig.node.json
new file mode 100644
index 000000000..413a56833
--- /dev/null
+++ b/apps/dashboard/tsconfig.node.json
@@ -0,0 +1,9 @@
+{
+  "compilerOptions": {
+    "allowSyntheticDefaultImports": true,
+    "composite": true,
+    "module": "ESNext",
+    "moduleResolution": "Node"
+  },
+  "include": ["vite.config.ts"]
+}
diff --git a/apps/dashboard/vite.config.ts b/apps/dashboard/vite.config.ts
new file mode 100644
index 000000000..134c849f1
--- /dev/null
+++ b/apps/dashboard/vite.config.ts
@@ -0,0 +1,19 @@
+import tailwindcss from "@tailwindcss/vite";
+import react from "@vitejs/plugin-react";
+import { defineConfig } from "vite";
+
+export default defineConfig({
+  build: {
+    sourcemap: true,
+    target: "esnext"
+  },
+  define: {
+    "process.env": {}
+  },
+  plugins: [react(), tailwindcss()],
+  server: {
+    host: true,
+    port: 5175,
+    strictPort: true
+  }
+});
diff --git a/bun.lock b/bun.lock
index 8518c1464..6247af59f 100644
--- a/bun.lock
+++ b/bun.lock
@@ -99,6 +99,27 @@
         "typescript": "catalog:",
       },
     },
+    "apps/dashboard": {
+      "name": "vortex-dashboard",
+      "version": "1.0.0",
+      "dependencies": {
+        "@tailwindcss/vite": "^4.0.3",
+        "@tanstack/react-query": "^5.64.2",
+        "@vitejs/plugin-react": "^4.3.4",
+        "clsx": "^2.1.1",
+        "react": "=19.2.0",
+        "react-dom": "=19.2.0",
+        "tailwind-merge": "^3.4.0",
+        "tailwindcss": "^4.0.3",
+      },
+      "devDependencies": {
+        "@types/node": "catalog:",
+        "@types/react": "^19.0.8",
+        "@types/react-dom": "^19.0.3",
+        "typescript": "catalog:",
+        "vite": "^6.2.6",
+      },
+    },
     "apps/frontend": {
       "name": "vortex-frontend",
       "version": "1.0.0",
@@ -4495,6 +4516,8 @@
 
     "vortex-backend": ["vortex-backend@workspace:apps/api"],
 
+    "vortex-dashboard": ["vortex-dashboard@workspace:apps/dashboard"],
+
     "vortex-frontend": ["vortex-frontend@workspace:apps/frontend"],
 
     "vortex-rebalancer": ["vortex-rebalancer@workspace:apps/rebalancer"],
diff --git a/package.json b/package.json
index 542b0c67e..2f7a3e1c5 100644
--- a/package.json
+++ b/package.json
@@ -101,15 +101,17 @@
   "packageManager": "bun@1.3.1",
   "private": true,
   "scripts": {
-    "build": "bun run build:shared && bun run build:sdk && bun run build:frontend && bun run build:backend",
+    "build": "bun run build:shared && bun run build:sdk && bun run build:frontend && bun run build:dashboard && bun run build:backend",
     "build:backend": "bun run --cwd apps/api build",
+    "build:dashboard": "bun run --cwd apps/dashboard build",
     "build:frontend": "bun run --cwd apps/frontend build",
     "build:sdk": "bun run --cwd packages/sdk build",
     "build:shared": "bun run --cwd packages/shared build",
     "compile:contracts:relayer": "bun run --cwd contracts/relayer compile",
-    "dev": "concurrently -n 'shared,backend,frontend' -c '#ffa500,#007755,#2f6da3' 'cd packages/shared && bun run dev' 'cd apps/api && bun dev' 'cd apps/frontend && bun dev'",
+    "dev": "concurrently -n 'shared,backend,frontend,dashboard' -c '#ffa500,#007755,#2f6da3,#24c989' 'cd packages/shared && bun run dev' 'cd apps/api && bun dev' 'cd apps/frontend && bun dev' 'cd apps/dashboard && bun dev'",
     "dev:backend": "bun run --cwd apps/api dev",
     "dev:contracts:relayer": "bun run --cwd contracts/relayer node",
+    "dev:dashboard": "bun run --cwd apps/dashboard dev",
     "dev:frontend": "bun run --cwd apps/frontend dev",
     "dev:rebalancer": "bun run --cwd apps/rebalancer dev",
     "docs:api:check": "bun docs/api/scripts/check-openapi.ts",
@@ -120,6 +122,7 @@
     "lint:fix": "biome lint --write .",
     "prepare": "husky",
     "serve:backend": "bun run --cwd apps/api serve",
+    "serve:dashboard": "bun run --cwd apps/dashboard preview",
     "serve:frontend": "bun run --cwd apps/frontend preview",
     "test:contracts:relayer": "bun run --cwd contracts/relayer test",
     "typecheck": "bunx tsc",

From 24d92abee03a8dab0fe1f3a41ccabf2237347d08 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 18:21:51 +0200
Subject: [PATCH 08/11] Fix dashboard local API access

---
 apps/api/package.json          | 2 +-
 apps/api/src/config/express.ts | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/apps/api/package.json b/apps/api/package.json
index b4037afe5..79a779fab 100644
--- a/apps/api/package.json
+++ b/apps/api/package.json
@@ -88,7 +88,7 @@
   "name": "vortex-backend",
   "scripts": {
     "build": "bun run swc src -d dist --strip-leading-paths",
-    "dev": "concurrently -n 'shared,backend' -c '#ffa500,007755' 'cd ../../packages/shared && bun run dev' 'bun --watch src/index.ts'",
+    "dev": "concurrently -n 'shared,backend' -c '#ffa500,007755' 'cd ../../packages/shared && bun run dev' 'NODE_ENV=development bun --watch src/index.ts'",
     "migrate": "bun -r @swc-node/register src/database/migrator.ts",
     "migrate:revert": "bun -r @swc-node/register src/database/migrator.ts revert-all",
     "migrate:revert-last": "bun -r @swc-node/register src/database/migrator.ts revert",
diff --git a/apps/api/src/config/express.ts b/apps/api/src/config/express.ts
index 5fc82df5c..61acb8ea0 100644
--- a/apps/api/src/config/express.ts
+++ b/apps/api/src/config/express.ts
@@ -35,6 +35,9 @@ app.use(
       "https://metrics.vortexfinance.co",
       config.env !== "production" ? "https://staging--vortexfi.netlify.app" : null,
       config.env === "development" ? "http://localhost:5173" : null,
+      config.env === "development" ? "http://127.0.0.1:5173" : null,
+      config.env === "development" ? "http://localhost:5175" : null,
+      config.env === "development" ? "http://127.0.0.1:5175" : null,
       config.env === "development" ? "http://localhost:6006" : null
     ].filter(Boolean) as string[]
   })

From ddf873bfad7234263bfe6ae799288304ad179c19 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 18:22:01 +0200
Subject: [PATCH 09/11] Document metrics dashboard access

---
 README.md                                     | 28 +++++++++++++++++++
 docs/security-spec/01-auth/admin-auth.md      |  2 +-
 .../07-operations/api-surface.md              |  4 +--
 .../07-operations/client-observability.md     |  7 +++++
 .../07-operations/secret-management.md        |  1 +
 5 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 303ba342e..6f46858c1 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,7 @@ This is a **Bun monorepo** containing multiple sub-projects organized into apps,
 ### Apps
 
 - **[apps/api](apps/api)** - Backend API service providing signature services, on/off-ramping flows, quote generation, and transaction state management
+- **[apps/dashboard](apps/dashboard)** - Internal React dashboard for protected API client observability data
 - **[apps/frontend](apps/frontend)** - React-based web application built with Vite for the Vortex user interface
 - **[apps/rebalancer](apps/rebalancer)** - Service for automated liquidity rebalancing across chains
 ### Contracts
@@ -67,6 +68,7 @@ bun dev
 
 This will start:
 - **Frontend**: [http://127.0.0.1:5173/](http://127.0.0.1:5173)
+- **Dashboard**: [http://127.0.0.1:5175/](http://127.0.0.1:5175)
 - **Backend API**: [http://localhost:3000](http://localhost:3000)
 
 #### Run Individual Projects
@@ -76,6 +78,11 @@ This will start:
 bun dev:frontend
 ```
 
+**Internal dashboard only:**
+```bash
+bun dev:dashboard
+```
+
 **Backend API only:**
 ```bash
 bun dev:backend
@@ -103,6 +110,9 @@ bun build
 # Build frontend
 bun build:frontend
 
+# Build internal dashboard
+bun build:dashboard
+
 # Build backend API
 bun build:backend
 
@@ -178,6 +188,24 @@ bun start
 
 See [apps/api/README.md](apps/api/README.md) for detailed API documentation.
 
+### Internal Dashboard (apps/dashboard)
+
+The internal dashboard displays sanitized API client observability events from `GET /v1/admin/api-client-events`. The backend endpoint requires `Authorization: Bearer <METRICS_DASHBOARD_SECRET>`, and the dashboard stores the entered token in browser session storage only. Use a different value than `ADMIN_SECRET` so a dashboard token compromise cannot access broader admin operations.
+
+**Development:**
+```bash
+cd apps/dashboard
+bun dev
+```
+
+**Build:**
+```bash
+cd apps/dashboard
+bun build
+```
+
+For Netlify, set the build command to `bun build` from `apps/dashboard` or use the root script `bun build:dashboard`. `VITE_DASHBOARD_API_BASE` can override the API base URL; otherwise the app uses `http://localhost:3000` in development and the `/api/<environment>` Netlify redirects in deployed environments.
+
 ### Rebalancer (apps/rebalancer)
 
 Service for automated liquidity rebalancing across chains.
diff --git a/docs/security-spec/01-auth/admin-auth.md b/docs/security-spec/01-auth/admin-auth.md
index 9729b221e..82cd463de 100644
--- a/docs/security-spec/01-auth/admin-auth.md
+++ b/docs/security-spec/01-auth/admin-auth.md
@@ -2,7 +2,7 @@
 
 ## What This Does
 
-Admin authentication protects internal/operational endpoints (partner management, system configuration, diagnostics). It uses a single shared secret (`ADMIN_SECRET` env var) compared via Bearer token.
+Admin authentication protects internal/operational endpoints that can mutate system state or manage partners. It uses a single shared secret (`ADMIN_SECRET` env var) compared via Bearer token. Read-only client observability dashboards use a separate `METRICS_DASHBOARD_SECRET` so a dashboard token compromise does not grant broader admin access.
 
 The flow:
 1. Admin includes `Authorization: Bearer <ADMIN_SECRET>` header
diff --git a/docs/security-spec/07-operations/api-surface.md b/docs/security-spec/07-operations/api-surface.md
index 56c1a6177..9bddae599 100644
--- a/docs/security-spec/07-operations/api-surface.md
+++ b/docs/security-spec/07-operations/api-surface.md
@@ -27,7 +27,7 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 - The API returns `X-Request-ID` so clients can include it in support/debug reports.
 - Partner-facing quote/ramp/auth outcomes are recorded as sanitized operational events; see `07-operations/client-observability.md`.
 
-**Route structure:** 27 route files under `api/routes/v1/`, each mounting controllers with appropriate auth middleware.
+**Route structure:** 27 TypeScript route files under `api/routes/v1/` including `index.ts`, each mounting controllers with appropriate auth middleware.
 
 ## Security Invariants
 
@@ -69,7 +69,7 @@ This spec covers the external-facing attack surface of the Vortex API (`apps/api
 - [N/A] Verify `NODE_ENV` is set to `"production"` in production — stack traces are only stripped when not in development mode. **N/A** — requires deployment configuration inspection.
 - [x] Verify error responses do not include internal error types, database error codes, or SQL fragments. **PASS** — error handler wraps errors in generic `APIError` format.
 - [x] Verify the `errors` array in `APIError` contains only user-facing messages, not internal field names or database column names. **PASS** — error messages are user-facing validation messages.
-- [x] Map all 27 route files and verify each has appropriate auth middleware (Supabase, API key, admin, or public). **PASS** — F-013 resolved (legacy `/pendulum/fundEphemeral`, `/moonbeam/execute-xcm`, `/subsidize/*` endpoints removed); `/v1/ramp/*` and `/v1/ramp/quotes(/best)` use `requirePartnerOrUserAuth()` with ownership guards; `/v1/brla/*` uses `requireAuth`; `/v1/mykobo/profiles` (GET + POST) use `requireAuth` (F-068 resolved); `/v1/maintenance/*` uses `adminAuth`; `/v1/webhook/*` uses `apiKeyAuth`.
+- [x] Map all 27 TypeScript route files and verify each has appropriate auth middleware (Supabase, API key, admin, metrics dashboard, or public). **PASS** — F-013 resolved (legacy `/pendulum/fundEphemeral`, `/moonbeam/execute-xcm`, `/subsidize/*` endpoints removed); `/v1/ramp/*` and `/v1/ramp/quotes(/best)` use `requirePartnerOrUserAuth()` with ownership guards; `/v1/brla/*` uses `requireAuth`; `/v1/mykobo/profiles` (GET + POST) use `requireAuth` (F-068 resolved); `/v1/maintenance/*` and `/v1/admin/partners/:partnerName/api-keys` use `adminAuth`; `/v1/admin/api-client-events` uses `metricsDashboardAuth`; `/v1/webhook/*` uses `apiKeyAuth`.
 - [x] Verify no route accidentally uses `publicKeyAuth` (public key only, no secret key) for operations that should require `apiKeyAuth` (secret key). **PASS** — auth middleware usage reviewed per route.
 - [N/A] Verify controllers do not pass raw `req.body` to database operations — check for Sequelize `.create(req.body)` or `.update(req.body)` patterns. **N/A** — deferred; requires comprehensive Sequelize usage audit.
 - [x] Verify no endpoint returns `process.env`, server config, or internal paths in responses. **PASS** — no endpoint exposes internal configuration.
diff --git a/docs/security-spec/07-operations/client-observability.md b/docs/security-spec/07-operations/client-observability.md
index 73e11ae36..06aaaae70 100644
--- a/docs/security-spec/07-operations/client-observability.md
+++ b/docs/security-spec/07-operations/client-observability.md
@@ -13,6 +13,8 @@ The observed surface includes:
 
 Events are persisted in `api_client_events` and structured logs are emitted through the existing backend logger. The event table is an operational telemetry store, not a source of truth for ramp state. Ramp execution failures remain in `RampState.errorLogs`; client observability events are request-level records used for dashboards, alerting, and incident investigation.
 
+Internal operators can inspect these events through `GET /v1/admin/api-client-events`, which is protected by the dedicated `Authorization: Bearer <METRICS_DASHBOARD_SECRET>` middleware. The Netlify-deployable dashboard in `apps/dashboard` calls this endpoint; it does not connect directly to the database and does not contain any server-side secrets. `METRICS_DASHBOARD_SECRET` must be different from `ADMIN_SECRET` to reduce blast radius.
+
 ## Security Invariants
 
 1. **Observability MUST NOT affect API behavior** — Event persistence, structured logging, and metric hooks must be best-effort. Failures in the observability layer must not change response bodies, HTTP statuses, ramp state, quote state, or retry behavior.
@@ -23,6 +25,7 @@ Events are persisted in `api_client_events` and structured logs are emitted thro
 6. **Partner attribution MUST use safe identifiers** — Events may store `partnerId`, `partnerName`, and short API key prefixes. Full secret keys and raw auth headers are forbidden.
 7. **Operational metrics MUST remain low-cardinality** — Future metric exporters must group by bounded labels such as operation, partner, status, HTTP status, and error type. They must not label by user ID, wallet address, request ID, quote ID, ramp ID, tax ID, PIX key, or free-form request values.
 8. **Event persistence SHOULD have automated retention before production alerting/dashboard rollout** — Raw operational events are useful for investigation but should not be retained indefinitely without aggregation or cleanup. Until that follow-up exists, operators should treat retention as a known operational gap.
+9. **Dashboard access MUST go through metrics-dashboard-authenticated backend APIs** — Browser dashboards must call protected backend endpoints and must not ship database credentials, Supabase service-role keys, Metabase embed secrets, or other server-only credentials to Netlify/frontend code.
 
 ## Threat Vectors & Mitigations
 
@@ -35,6 +38,8 @@ Events are persisted in `api_client_events` and structured logs are emitted thro
 | **Missing correlation during incidents** — Operators cannot connect a partner report to backend logs | Generate or propagate `requestId` for all requests and return it via `X-Request-ID`. Persist request IDs alongside quote/ramp IDs when available. |
 | **High-cardinality metric explosion** — Future dashboard metrics use ramp IDs or user IDs as labels | Keep high-cardinality identifiers in logs/event rows only. Export aggregate metrics using bounded labels. |
 | **Unbounded telemetry retention** — Raw event rows grow indefinitely | Known follow-up: add retention or aggregation before long-term production alerting/dashboard operation. Initial raw retention should be time-bounded, ideally 30-90 days. |
+| **Public dashboard exposure** — A static dashboard URL is discovered by outsiders | Require the dedicated backend metrics dashboard bearer token for all event data. Do not rely on obscurity of Netlify URLs. Keep frontend tokens in operator-controlled browser session storage only. |
+| **BI embed secret leak** — A future Metabase embed is generated in browser code | Generate signed embed URLs only from the backend. Do not place Metabase signing secrets in Netlify environment variables exposed to Vite. |
 
 ## Audit Checklist
 
@@ -46,4 +51,6 @@ Events are persisted in `api_client_events` and structured logs are emitted thro
 - [ ] Verify no observability event stores `X-API-Key`, bearer tokens, raw headers, raw request bodies, tax IDs, PIX destinations, QR codes, KYC data, private keys, seeds, ephemeral secrets, or signed transaction payloads.
 - [ ] Verify error messages are truncated and dashboards/alerts use stable `errorType` categories rather than raw messages.
 - [ ] Verify future metric exporters do not use request ID, quote ID, ramp ID, user ID, wallet address, tax ID, or PIX key as metric labels.
+- [ ] Verify `GET /v1/admin/api-client-events` uses `metricsDashboardAuth` and returns only sanitized event fields.
+- [ ] Verify `apps/dashboard` has no direct database connection and no server-only credentials in Vite-exposed env vars.
 - [ ] Add and verify an automated retention or aggregation mechanism before retaining high-volume production telemetry long-term.
diff --git a/docs/security-spec/07-operations/secret-management.md b/docs/security-spec/07-operations/secret-management.md
index efac1a7c9..e361eeceb 100644
--- a/docs/security-spec/07-operations/secret-management.md
+++ b/docs/security-spec/07-operations/secret-management.md
@@ -18,6 +18,7 @@ This spec catalogs every secret, its purpose, its blast radius if compromised, a
 | `MOONBEAM_FUNDING_PRIVATE_KEY` | EVM subsidization transfers across all EVM chains in scope (Moonbeam, Base, Polygon, etc.); BRLA payouts on Base; EVM fee distribution on Base | Drain of EVM funding pool on every supported EVM chain — including BRLA payout path on Base |
 | `CLIENT_DOMAIN_SECRET` | SEP-10 domain signing for Stellar anchors | Impersonation of Vortex in Stellar anchor authentication |
 | `ADMIN_SECRET` | Admin endpoint bearer token | Full admin access — can modify ramps, trigger operations |
+| `METRICS_DASHBOARD_SECRET` | Internal observability dashboard bearer token | Read-only access to sanitized API client event data |
 | `WEBHOOK_PRIVATE_KEY` | RSA key for webhook signatures | Forge webhook signatures — could trick consumers into accepting fake events. **If missing, ephemeral RSA keys are generated at startup (non-persistent across restarts).** |
 | `SUPABASE_SERVICE_KEY` | Supabase admin access (bypasses RLS) | Full database read/write — all ramp data, user data, keys |
 | `SUPABASE_ANON_KEY` | Supabase public access (subject to RLS) | Limited by RLS policies — lower blast radius than service key |

From fe6d6f9782ca8f64e8dea3f2271a0aea570d5908 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 19:01:36 +0200
Subject: [PATCH 10/11] Address code review comments

---
 apps/api/src/api/middlewares/apiKeyAuth.ts    | 10 ++++--
 .../apiClientEvent.service.test.ts            | 13 +++++++
 .../observability/apiClientEvent.service.ts   |  1 +
 apps/api/src/api/observability/types.ts       |  1 +
 apps/api/src/config/express.ts                |  1 +
 apps/dashboard/src/App.tsx                    |  9 ++---
 .../dashboard/src/api/apiClientEvents.test.ts | 35 ++++++++++++++++++-
 apps/dashboard/src/api/apiClientEvents.ts     | 11 +++++-
 8 files changed, 72 insertions(+), 9 deletions(-)

diff --git a/apps/api/src/api/middlewares/apiKeyAuth.ts b/apps/api/src/api/middlewares/apiKeyAuth.ts
index adb15bde3..6047fb808 100644
--- a/apps/api/src/api/middlewares/apiKeyAuth.ts
+++ b/apps/api/src/api/middlewares/apiKeyAuth.ts
@@ -104,7 +104,7 @@ export function apiKeyAuth(options: ApiKeyAuthOptions = {}) {
           const requestedPartner = await Partner.findByPk(partnerIdOrName);
 
           if (!requestedPartner) {
-            recordAuthFailure(req, 404, "auth_invalid_public_key", getSafeKeyPrefix(apiKey), partner);
+            recordAuthFailure(req, 404, "auth_partner_not_found", getSafeKeyPrefix(apiKey), partner);
             return res.status(404).json({
               error: {
                 code: "PARTNER_NOT_FOUND",
@@ -180,7 +180,7 @@ export function enforcePartnerAuth() {
         const requestedPartner = await Partner.findByPk(partnerIdOrName);
 
         if (!requestedPartner) {
-          recordAuthFailure(req, 404, "auth_invalid_public_key", null, req.authenticatedPartner);
+          recordAuthFailure(req, 404, "auth_partner_not_found", null, req.authenticatedPartner);
           return res.status(404).json({
             error: {
               code: "PARTNER_NOT_FOUND",
@@ -222,7 +222,11 @@ function recordAuthFailure(
   httpStatus: number,
   errorType: Extract<
     ApiClientErrorType,
-    "auth_missing_api_key" | "auth_invalid_api_key" | "auth_invalid_public_key" | "auth_partner_mismatch"
+    | "auth_missing_api_key"
+    | "auth_invalid_api_key"
+    | "auth_invalid_public_key"
+    | "auth_partner_not_found"
+    | "auth_partner_mismatch"
   >,
   apiKeyPrefix?: string | null,
   partner?: AuthenticatedPartner
diff --git a/apps/api/src/api/observability/apiClientEvent.service.test.ts b/apps/api/src/api/observability/apiClientEvent.service.test.ts
index dc0ced211..b1c4f029b 100644
--- a/apps/api/src/api/observability/apiClientEvent.service.test.ts
+++ b/apps/api/src/api/observability/apiClientEvent.service.test.ts
@@ -34,6 +34,19 @@ describe("sanitizeApiClientEvent", () => {
     expect(sanitized.errorType).toBe("none");
     expect(sanitized.errorMessage).toBeNull();
   });
+
+  it("uses a distinct safe message for missing partner records", () => {
+    const sanitized = sanitizeApiClientEvent({
+      errorMessage: "Partner 123e4567-e89b-12d3-a456-426614174000 was not found",
+      errorType: "auth_partner_not_found",
+      operation: "auth_api_key",
+      status: "failure"
+    });
+
+    expect(sanitized.errorType).toBe("auth_partner_not_found");
+    expect(sanitized.errorMessage).toBe("Requested partner was not found.");
+    expect(sanitized.errorMessage).not.toContain("123e4567-e89b-12d3-a456-426614174000");
+  });
 });
 
 describe("recordApiClientEventSafe", () => {
diff --git a/apps/api/src/api/observability/apiClientEvent.service.ts b/apps/api/src/api/observability/apiClientEvent.service.ts
index 91910b693..a8b850485 100644
--- a/apps/api/src/api/observability/apiClientEvent.service.ts
+++ b/apps/api/src/api/observability/apiClientEvent.service.ts
@@ -80,6 +80,7 @@ function getSafeErrorMessage(errorType: ApiClientErrorType): string | null {
     auth_invalid_public_key: "Public API key validation failed.",
     auth_missing_api_key: "API authentication is missing.",
     auth_partner_mismatch: "Authenticated partner does not match the requested partner.",
+    auth_partner_not_found: "Requested partner was not found.",
     internal_error: "Internal server error.",
     invalid_ephemerals: "Ephemeral account validation failed.",
     invalid_presigned_transactions: "Presigned transaction validation failed.",
diff --git a/apps/api/src/api/observability/types.ts b/apps/api/src/api/observability/types.ts
index b4f9f4fb3..60ac1ab6a 100644
--- a/apps/api/src/api/observability/types.ts
+++ b/apps/api/src/api/observability/types.ts
@@ -20,6 +20,7 @@ export type ApiClientErrorType =
   | "auth_missing_api_key"
   | "auth_invalid_api_key"
   | "auth_invalid_public_key"
+  | "auth_partner_not_found"
   | "auth_partner_mismatch"
   | "auth_inactive_partner"
   | "ownership_denied"
diff --git a/apps/api/src/config/express.ts b/apps/api/src/config/express.ts
index 61acb8ea0..f8c7aebfa 100644
--- a/apps/api/src/config/express.ts
+++ b/apps/api/src/config/express.ts
@@ -28,6 +28,7 @@ app.use(
   cors({
     allowedHeaders: ["Content-Type", "Authorization", "X-API-Key", "X-Request-ID", "X-Correlation-ID"],
     credentials: true,
+    exposedHeaders: ["X-Request-ID"],
     maxAge: 86400, // Cache preflight requests for 24 hours
     methods: "GET,HEAD,PUT,PATCH,POST,DELETE", // Explicitly list allowed headers
     origin: [
diff --git a/apps/dashboard/src/App.tsx b/apps/dashboard/src/App.tsx
index 819dba10d..149dfd7e3 100644
--- a/apps/dashboard/src/App.tsx
+++ b/apps/dashboard/src/App.tsx
@@ -1,6 +1,6 @@
 import { useQuery, useQueryClient } from "@tanstack/react-query";
 import { type FormEvent, type ReactNode, useMemo, useState } from "react";
-import { ApiClientEventsFilters, fetchApiClientEvents } from "./api/apiClientEvents";
+import { ApiClientEventsFilters, fetchApiClientEvents, normalizeMetricsDashboardToken } from "./api/apiClientEvents";
 import { cn } from "./helpers/cn";
 
 const tokenStorageKey = "vortex-dashboard-metrics-token";
@@ -51,9 +51,10 @@ export function App() {
 
   function handleTokenSubmit(event: FormEvent<HTMLFormElement>) {
     event.preventDefault();
-    const trimmedToken = tokenDraft.trim();
-    window.sessionStorage.setItem(tokenStorageKey, trimmedToken);
-    setMetricsToken(trimmedToken);
+    const normalizedToken = normalizeMetricsDashboardToken(tokenDraft);
+    window.sessionStorage.setItem(tokenStorageKey, normalizedToken);
+    setMetricsToken(normalizedToken);
+    setTokenDraft(normalizedToken);
   }
 
   function handleFiltersSubmit(event: FormEvent<HTMLFormElement>) {
diff --git a/apps/dashboard/src/api/apiClientEvents.test.ts b/apps/dashboard/src/api/apiClientEvents.test.ts
index db89d6817..80232ef42 100644
--- a/apps/dashboard/src/api/apiClientEvents.test.ts
+++ b/apps/dashboard/src/api/apiClientEvents.test.ts
@@ -1,5 +1,5 @@
 import { afterEach, describe, expect, it, mock } from "bun:test";
-import { buildApiClientEventsUrl, fetchApiClientEvents } from "./apiClientEvents";
+import { buildApiClientEventsUrl, fetchApiClientEvents, normalizeMetricsDashboardToken } from "./apiClientEvents";
 
 describe("buildApiClientEventsUrl", () => {
   it("builds relative Netlify proxy URLs with query filters", () => {
@@ -40,4 +40,37 @@ describe("fetchApiClientEvents", () => {
       }
     });
   });
+
+  it("normalizes a pasted bearer token before sending the auth header", async () => {
+    const fetchMock = mock(async (_input: string | URL | Request, _init?: RequestInit) =>
+      Response.json({
+        events: [],
+        limit: 50,
+        offset: 0,
+        summary: { byErrorType: {}, byOperation: {}, byStatus: {}, sampleSize: 0, total: 0 },
+        total: 0
+      })
+    );
+    globalThis.fetch = fetchMock as unknown as typeof fetch;
+
+    await fetchApiClientEvents({ limit: 50, offset: 0 }, "Bearer metrics-dashboard-secret");
+
+    const firstFetchCall = fetchMock.mock.calls[0];
+    expect(firstFetchCall).toBeDefined();
+
+    const init = firstFetchCall?.[1];
+    expect(init).toEqual({
+      headers: {
+        Authorization: "Bearer metrics-dashboard-secret"
+      }
+    });
+  });
+});
+
+describe("normalizeMetricsDashboardToken", () => {
+  it("accepts raw token values and pasted bearer header values", () => {
+    expect(normalizeMetricsDashboardToken("metrics-dashboard-secret")).toBe("metrics-dashboard-secret");
+    expect(normalizeMetricsDashboardToken("Bearer metrics-dashboard-secret")).toBe("metrics-dashboard-secret");
+    expect(normalizeMetricsDashboardToken(" bearer   metrics-dashboard-secret ")).toBe("metrics-dashboard-secret");
+  });
 });
diff --git a/apps/dashboard/src/api/apiClientEvents.ts b/apps/dashboard/src/api/apiClientEvents.ts
index 39ceb9870..fff598ed3 100644
--- a/apps/dashboard/src/api/apiClientEvents.ts
+++ b/apps/dashboard/src/api/apiClientEvents.ts
@@ -49,6 +49,13 @@ export type ApiClientEventsResponse = {
   total: number;
 };
 
+export function normalizeMetricsDashboardToken(token: string): string {
+  return token
+    .trim()
+    .replace(/^Bearer\s+/i, "")
+    .trim();
+}
+
 export function buildApiClientEventsUrl(filters: ApiClientEventsFilters, apiBaseUrl = config.apiBaseUrl): string {
   const params = new URLSearchParams();
 
@@ -68,9 +75,11 @@ export async function fetchApiClientEvents(
   filters: ApiClientEventsFilters,
   metricsDashboardToken: string
 ): Promise<ApiClientEventsResponse> {
+  const normalizedToken = normalizeMetricsDashboardToken(metricsDashboardToken);
+
   const response = await fetch(buildApiClientEventsUrl(filters), {
     headers: {
-      Authorization: `Bearer ${metricsDashboardToken}`
+      Authorization: `Bearer ${normalizedToken}`
     }
   });
 

From 4210e1968accdee1ffa2856c0a1d2a518304fc91 Mon Sep 17 00:00:00 2001
From: Marcel Ebert <mail@marcel-ebert.de>
Date: Fri, 5 Jun 2026 21:45:21 +0200
Subject: [PATCH 11/11] Remove backend-client-observability-plan.md

---
 .../backend-client-observability-plan.md      | 561 ------------------
 1 file changed, 561 deletions(-)
 delete mode 100644 .sisyphus/plans/backend-client-observability-plan.md

diff --git a/.sisyphus/plans/backend-client-observability-plan.md b/.sisyphus/plans/backend-client-observability-plan.md
deleted file mode 100644
index a9c889d60..000000000
--- a/.sisyphus/plans/backend-client-observability-plan.md
+++ /dev/null
@@ -1,561 +0,0 @@
-# Backend Client Observability Implementation Plan
-
-## Context
-
-We want backend-side observability for Vortex partner/API-client issues before adding alerts and dashboards. The immediate goal is to record reliable metrics and structured logs for partner-facing API flows without changing existing quote/ramp behavior.
-
-Primary flows to observe:
-
-- Quote creation and retrieval
-- Ramp register
-- Ramp update
-- Ramp start
-- Ramp status
-- Ramp error log retrieval
-- Auth/config failures around these flows
-
-The monitoring should help answer:
-
-- Which partner/API client is having issues?
-- Which operation is failing?
-- Is the issue auth/config, validation, expired quote, missing presigned transactions, backend error, or async ramp execution?
-- Is this isolated to one partner or global?
-- Which request IDs, quote IDs, and ramp IDs can be used for debugging?
-
-## Core Design Principle
-
-Observability must run alongside existing flows and must not affect business behavior.
-
-Rules:
-
-1. Do not change controller/service response bodies.
-2. Do not change HTTP statuses.
-3. Do not introduce retries or new side effects in quote/ramp flows.
-4. Do not block API responses on metric/log persistence.
-5. Do not allow observability failures to throw into request handling.
-6. Do not log secrets or sensitive user/payment/KYC data.
-
-All observability writes must be best-effort. If persistence fails, the API request should continue exactly as it does today.
-
-## Persistence Decision
-
-### Recommendation
-
-Use an append-only database table for persistent operational events, plus existing Winston logs for detailed application logging. Do not use Supabase Storage as the primary metrics store.
-
-### Why not only in-memory metrics?
-
-In-memory counters reset on server restart and are not enough if we want historical partner health, later dashboarding, or investigation across deployments. They are still useful for Prometheus-style scraping later, but they should not be the only source of partner health history.
-
-### Supabase Storage bucket option
-
-Supabase Storage is better for raw archive files, such as daily NDJSON log exports. It is not ideal as the primary backend monitoring store because:
-
-- querying by partner, operation, status, and time window is awkward
-- alert/dashboard queries require downloading/parsing files or secondary indexing
-- concurrent append patterns are less natural than database inserts
-- retention and aggregation become custom jobs
-
-Storage can be added later as a cold archive, but not as the main operational source.
-
-### Extra table option
-
-An append-only table is better for this phase because:
-
-- it survives restarts
-- it supports per-partner dashboard queries
-- it supports alert queries over time windows
-- it is easy to join conceptually with partner/quote/ramp identifiers
-- it can be indexed by `createdAt`, `partnerId`, `operation`, `status`, and `errorType`
-
-### Important caveat
-
-Do not write one table row for every noisy low-value event forever without retention. Start with partner-facing operations only, and add retention/aggregation before traffic grows significantly.
-
-Recommended initial retention:
-
-- raw operational events: 30-90 days
-- later aggregated daily/hourly summaries: longer retention if needed
-
-## Proposed Data Model
-
-Create a new Sequelize model and migration for an append-only table, tentatively named `ApiClientEvent`.
-
-Suggested table: `api_client_events`
-
-Fields:
-
-```text
-id UUID primary key
-request_id string nullable/indexed
-operation string indexed
-status string indexed                 # success | failure
-http_status integer nullable indexed
-error_type string nullable indexed
-error_message string nullable          # sanitized, short message only
-partner_id UUID nullable indexed
-partner_name string nullable indexed
-api_key_prefix string nullable indexed # never secret key value
-user_id UUID nullable indexed
-quote_id UUID nullable indexed
-ramp_id UUID nullable indexed
-ramp_type string nullable
-network string nullable
-payment_method string nullable
-duration_ms integer nullable
-metadata jsonb nullable                # sanitized low-volume details only
-created_at timestamp indexed
-updated_at timestamp
-```
-
-Notes:
-
-- `quote_id` and `ramp_id` are acceptable in the table, but should not be metric labels.
-- `error_message` must be sanitized and short. Prefer stable `error_type` for dashboards.
-- `metadata` must never contain request bodies, raw headers, tax IDs, PIX keys, QR codes, KYC data, API secrets, or ephemeral secrets.
-- Use partner name/ID and API key prefix only for attribution.
-
-## Metrics Shape
-
-Persistent events support historical querying. In addition, expose or prepare metric-style counters/histograms through a small abstraction so a Prometheus/Grafana or Datadog integration can be added later.
-
-Metric names to prepare:
-
-```text
-vortex_api_client_requests_total{operation,partner,status,http_status,error_type}
-vortex_api_client_request_duration_seconds{operation,partner,status}
-vortex_api_auth_failures_total{route,auth_error_type,partner}
-vortex_ramp_operation_failures_total{partner,operation,error_type}
-```
-
-Metric label rules:
-
-- OK: partner slug/name/id, operation, status, HTTP status, error type, environment
-- Not OK: ramp ID, quote ID, user ID, wallet address, tax ID, PIX key, request ID
-
-High-cardinality identifiers belong in structured logs and the persistent event table, not labels.
-
-## Structured Logging Shape
-
-Use the existing backend logger. Add one structured warning/error log for failed partner-facing operations and optionally one compact info log for successful operations.
-
-Failure log shape:
-
-```ts
-logger.warn("Partner API operation failed", {
-  requestId,
-  operation,
-  partnerId,
-  partnerName,
-  apiKeyPrefix,
-  userId,
-  quoteId,
-  rampId,
-  httpStatus,
-  errorType,
-  errorMessage,
-  durationMs
-});
-```
-
-Success log shape:
-
-```ts
-logger.info("Partner API operation completed", {
-  requestId,
-  operation,
-  partnerId,
-  partnerName,
-  apiKeyPrefix,
-  userId,
-  quoteId,
-  rampId,
-  durationMs
-});
-```
-
-Logging guidance:
-
-- Always log failures.
-- Success logs can be sampled or omitted later if volume is too high.
-- Never log raw request bodies or headers.
-- Never log API secret keys, KYC data, PIX data, QR codes, tax IDs, or ephemeral secret material.
-
-## Error Classification
-
-Add a helper that maps known errors into stable `error_type` values. Dashboards and alerts should use these stable categories instead of raw error messages.
-
-Initial categories:
-
-```text
-none
-validation_error
-auth_missing_api_key
-auth_invalid_api_key
-auth_invalid_public_key
-auth_partner_mismatch
-auth_inactive_partner
-ownership_denied
-quote_not_found
-quote_expired
-quote_consumed
-invalid_ephemerals
-invalid_presigned_transactions
-missing_presigned_transactions
-time_window_exceeded
-ramp_not_found
-ramp_not_updatable
-ramp_not_in_initial_state
-service_unavailable
-provider_error
-internal_error
-unknown_error
-```
-
-Classification should be conservative. If unsure, classify as `unknown_error` or `internal_error` rather than parsing sensitive details.
-
-## Proposed Module Layout
-
-Add a small observability module under the API app:
-
-```text
-apps/api/src/api/observability/
-  requestContext.ts
-  apiClientEvents.model.ts        # or place model under apps/api/src/models/
-  apiClientEvent.service.ts
-  metrics.ts
-  operationLogger.ts
-  errorClassifier.ts
-  types.ts
-```
-
-Preferred model location should follow existing repo patterns. Since existing Sequelize models live in `apps/api/src/models`, the actual model should likely be:
-
-```text
-apps/api/src/models/apiClientEvent.model.ts
-```
-
-and exported from:
-
-```text
-apps/api/src/models/index.ts
-```
-
-Responsibilities:
-
-- `requestContext.ts`: generate/propagate request ID and timing context.
-- `apiClientEvent.service.ts`: best-effort event persistence.
-- `metrics.ts`: safe counters/histogram helpers or placeholders for future exporter integration.
-- `operationLogger.ts`: structured success/failure logs.
-- `errorClassifier.ts`: stable error category mapping.
-- `types.ts`: shared operation/status/event types.
-
-Every public observability function should catch and swallow its own failures after logging a local debug/warn message if appropriate.
-
-## Request Context Middleware
-
-Add middleware early in Express setup to:
-
-- read an incoming request/correlation ID header if present
-- generate a request ID if absent
-- attach it to `req`
-- add `X-Request-ID` to the response
-- record `startedAt` for duration calculation
-
-Candidate headers to accept:
-
-```text
-X-Request-ID
-X-Correlation-ID
-```
-
-If both exist, prefer `X-Request-ID`.
-
-The middleware must not require any new client behavior.
-
-## Instrumentation Points
-
-### Auth/config failures
-
-Instrument auth-related failure branches in:
-
-- `apps/api/src/api/middlewares/apiKeyAuth.ts`
-- `apps/api/src/api/middlewares/publicKeyAuth.ts`
-- `apps/api/src/api/middlewares/dualAuth.ts`
-- `apps/api/src/api/middlewares/ownershipAuth.ts`
-
-Record events for:
-
-- missing API key
-- invalid API key format
-- invalid API key
-- invalid public key
-- partner mismatch
-- inactive/expired key if distinguishable
-- ownership denied
-
-These events may not have quote/ramp context yet. Use route, partner if known, key prefix if safe, and request ID.
-
-### Quote controller
-
-Instrument:
-
-- `createQuote`
-- `createBestQuote`
-- `getQuote`
-
-Relevant files:
-
-- `apps/api/src/api/controllers/quote.controller.ts`
-- `apps/api/src/api/services/quote/index.ts`
-- quote finalization/persistence service where `QuoteTicket` is created
-
-Capture:
-
-- operation
-- partner ID/name if known
-- public API key or prefix if already stored as safe public key
-- quote ID on success
-- ramp type
-- network
-- payment method
-- duration
-- status/error type
-
-### Ramp controller/service
-
-Instrument:
-
-- `registerRamp`
-- `updateRamp`
-- `startRamp`
-- `getRampStatus`
-- `getRampErrors`
-
-Relevant files:
-
-- `apps/api/src/api/controllers/ramp.controller.ts`
-- `apps/api/src/api/services/ramp/ramp.service.ts`
-
-Capture:
-
-- operation
-- partner ID/name from authenticated request if available
-- partner ID/name via `RampState -> QuoteTicket` or `QuoteTicket` after records are loaded
-- quote ID
-- ramp ID
-- ramp type if available
-- network/payment method if available
-- duration
-- status/error type
-
-Do not alter existing service behavior. Add observation around existing success/catch paths only.
-
-### Async phase execution
-
-Do not mix async phase execution failures with synchronous partner API call failures in the first pass.
-
-Existing phase errors are already stored in `RampState.errorLogs`. Later, phase failures can emit their own separate events such as:
-
-```text
-operation = ramp_phase_execution
-phase = nablaSwap | performBrlaPayout | ...
-```
-
-For this initial plan, focus on partner-facing request operations.
-
-## Event Recording Pattern
-
-Use a best-effort helper instead of direct model writes in controllers.
-
-Example shape:
-
-```ts
-await recordApiClientEventSafe({
-  requestId,
-  operation: "ramp_start",
-  status: "failure",
-  httpStatus,
-  errorType,
-  partnerId,
-  partnerName,
-  quoteId,
-  rampId,
-  durationMs
-});
-```
-
-The helper must internally catch errors:
-
-```ts
-export async function recordApiClientEventSafe(event: ApiClientEventInput): Promise<void> {
-  try {
-    await ApiClientEvent.create(sanitizeEvent(event));
-  } catch (error) {
-    logger.warn("Failed to record API client event", { error: error instanceof Error ? error.message : String(error) });
-  }
-}
-```
-
-If we are concerned about adding DB write latency to the request path, make the helper fire-and-forget:
-
-```ts
-void recordApiClientEventSafe(event);
-```
-
-Preferred initial approach:
-
-- use fire-and-forget persistence for event rows
-- make observability failures impossible to surface to clients
-- keep event payload small
-
-Tradeoff: fire-and-forget can lose events if the process exits immediately. That is acceptable for non-critical observability and safer than blocking ramp flows.
-
-## Migration and Model Plan
-
-1. Add Sequelize migration for `api_client_events`.
-2. Add `ApiClientEvent` model.
-3. Export model from `apps/api/src/models/index.ts`.
-4. Add indexes:
-   - `created_at`
-   - `partner_id, created_at`
-   - `partner_name, created_at`
-   - `operation, created_at`
-   - `status, created_at`
-   - `error_type, created_at`
-   - `request_id`
-   - `quote_id`
-   - `ramp_id`
-5. Consider retention cleanup in a later migration/worker, not in the first implementation unless there is an existing cleanup pattern that makes it trivial.
-
-## Rollout Steps
-
-### Step 1: Foundation
-
-- Add request context middleware.
-- Add request ID response header.
-- Add observability types, error classifier, event service, and structured logging helper.
-- Add model/migration for `api_client_events`.
-
-Verification:
-
-- Existing API tests pass.
-- Manual request receives `X-Request-ID`.
-- No response body/status changes.
-
-### Step 2: Auth events
-
-- Instrument auth/config failure branches.
-- Record persistent events and structured logs for failures only.
-
-Verification:
-
-- Existing auth failures still return the same status/body.
-- Event rows are created for invalid/missing keys.
-- No secret key is logged or stored.
-
-### Step 3: Quote events
-
-- Instrument quote create/best/get success and failure.
-- Persist compact events.
-- Log failures.
-
-Verification:
-
-- Existing quote behavior unchanged.
-- Success/failure rows include operation, partner when known, quote ID on success, duration.
-
-### Step 4: Ramp request events
-
-- Instrument ramp register/update/start/status/errors success and failure.
-- Enrich partner attribution from `QuoteTicket`/`RampState` where available.
-
-Verification:
-
-- Existing ramp behavior unchanged.
-- Event rows include ramp/quote IDs where available.
-- Expected error categories are stable.
-
-### Step 5: Metric abstraction
-
-- Add safe metric helper functions mirroring event writes.
-- Keep implementation simple initially: either no-op counters plus persistent events, or in-process counters if a metrics endpoint already exists.
-- Do not block on full Prometheus/Grafana setup in this phase.
-
-Verification:
-
-- Metric helper calls cannot throw.
-- Label values are low-cardinality.
-
-### Step 6: Follow-up foundation for alerts/dashboard
-
-- Add simple query helpers for partner health if needed.
-- Example queries:
-  - failures by partner/operation over last 15 minutes
-  - zero success despite traffic
-  - top error types by partner over last 24 hours
-  - recent failed ramp IDs for partner
-
-This step can be done just before implementing alerts/dashboard.
-
-## Testing Plan
-
-Add focused tests where practical:
-
-- `errorClassifier` maps known errors/statuses to stable categories.
-- `sanitizeEvent` removes or rejects sensitive fields.
-- `recordApiClientEventSafe` swallows persistence errors.
-- request context middleware sets request ID and response header.
-
-Regression checks:
-
-- Existing quote tests pass.
-- Existing ramp tests pass.
-- Existing auth tests pass, if present.
-- `bun lint:fix`
-- `bun typecheck`
-- `bun build:backend`
-
-Follow repo guidance:
-
-- Always use `bun`.
-- Run `bun lint:fix` after code changes.
-- If shared package changes are made, run `bun build:shared` first. This plan should avoid shared changes unless necessary.
-
-## Privacy and Safety Checklist
-
-Before merging implementation, verify:
-
-- No `secretKey` or `X-API-Key` values are stored/logged.
-- No raw request headers are stored/logged.
-- No raw request bodies are stored/logged.
-- No tax IDs are stored/logged.
-- No PIX keys/destinations are stored/logged.
-- No QR codes/payment payloads are stored/logged.
-- No KYC data is stored/logged.
-- No ephemeral private keys or signed transaction payloads are stored/logged.
-- Metrics labels do not include request ID, quote ID, ramp ID, user ID, wallet addresses, or any free-form user data.
-- Observability write failures cannot affect API responses.
-
-## Open Questions for Implementation Time
-
-Resolve these by reading current code before editing:
-
-1. Exact migration naming/date convention in `apps/api/src/database/migrations`.
-2. Whether a `/metrics` endpoint or Prometheus client already exists.
-3. Exact Express request type augmentation pattern in this repo.
-4. Existing tests around quote/ramp controllers and whether event persistence should be mocked.
-5. Whether success events should be recorded for all operations initially, or only failures plus aggregate counters.
-
-Default answer for item 5: record both success and failure for partner-facing operations initially, because the dashboard needs denominators. If volume becomes an issue, add retention/aggregation rather than removing success visibility prematurely.
-
-## Success Criteria
-
-The implementation is complete when:
-
-- Partner-facing quote/ramp/auth operations create persistent sanitized operational events.
-- Failures create structured logs with request ID, operation, partner attribution when available, stable error type, and duration.
-- Existing API behavior is unchanged.
-- Event persistence failures are swallowed and cannot break quote/ramp flows.
-- Sensitive fields are not stored or logged.
-- The resulting table can answer per-partner failure-rate and recent-failure queries needed for future alerts/dashboard work.