POST /v1/auth: p90 3.8 s, p99 13.4 s — no keep-alive and no timeout on Monero signature verification

[Danswar]

POST /v1/auth has a heavy latency tail in production: p50 109 ms but p90 3.8 s, p99 13.4 s, max 15.7 s (n=138 over 3 h; 38 calls over 1 s, 7 over 8 s). A TraceQL search for auth spans over 2 s returns the search limit (50) for a 6 h window, with individual calls up to 20 s.

The tail is not caused by the database. It has two contributors: a misconfigured outbound HTTP call in the Monero verification path, and a chronically saturated event loop.

1. Monero signature verification: new agent per call, no keep-alive, no timeout

src/integration/blockchain/monero/monero-client.ts:398-404

private httpConfig(): HttpRequestConfig {
  return {
    httpsAgent: new Agent({
      ca: Config.blockchain.monero.certificate,
    }),
  };
}

httpConfig() is invoked inline on every request (monero-client.ts:208-219), so a brand-new https.Agent is constructed per call. keepAlive is not set, so Node's default false applies and the socket is closed after each response. Both problems compound: even with keepAlive: true there would be no cross-call socket pool to reuse. grep -rn "keepAlive" src/ returns zero hits repo-wide, and HttpModule is imported bare in src/shared/shared.module.ts:46, so there is no global agent either.

Result: a full TCP + TLS handshake on every Monero sign-in. Measured tcp.connect durations to the Monero wallet-RPC host in production traces: 209 ms, 418 ms, 1013 ms, 1370 ms.

There is also no timeout. httpConfig() sets only httpsAgent, the Monero config block (src/config/config.ts:1075-1084) has no timeout field, and HttpModule is registered without options, so axios' default timeout: 0 applies — wait forever. Other integrations do set one (config.ts:1240 exchanges timeout: 30000, config.ts:1015 CLEMENTINE_TIMEOUT_MS), so this is a Monero-specific omission rather than house style. A hung wallet-RPC socket pins the request indefinitely.

The cost is paid up to twice per sign-in — src/subdomains/generic/user/models/auth/auth.service.ts:559-561:

let isValid = await this.cryptoService.verifySignature(defaultMessage, address, signature, key, blockchain);
if (!isValid)
  isValid = await this.cryptoService.verifySignature(fallbackMessage, address, signature, key, blockchain);

src/integration/blockchain/zano/zano-client.ts:53-65 has the same missing timeout (it passes no agent at all, so it at least uses axios' shared default agent).

2. Event-loop saturation

MonitorEventLoopService, production, 6 h window (2155 samples):

metric	value
utilization p50	95 %
samples above 90 %	1461 / 2155 (68 %)
delay p95	2042 ms
delay max	5352 ms

With the loop blocked, every async callback is delayed. In a representative 4597 ms auth trace:

span	duration
pg SELECT (user lookup)	461 ms
pg START TRANSACTION	458 ms
pg INSERT user_data	2060 ms
pg START TRANSACTION	587 ms

START TRANSACTION does essentially no work server-side; 458 ms and 587 ms for it are scheduling delay, not database time. Other traces show dns.lookup of an internal service name at 2180 ms, tcp.connect at 4009 ms, and pg.connect at 7997 ms — again, these are callbacks that could not be scheduled, not network faults.

The database is not the bottleneck

Three independent checks:

1. The 2060 ms INSERT INTO "user_data" span does not appear in the database's own slow-query log, which logged a 117.5 ms statement in the same window — so the logging threshold is at most 117 ms and the insert executed well below it.
2. START TRANSACTION at 458 ms / 587 ms client-side, absent from the database log entirely.
3. The connection-pool watchdog (src/subdomains/core/monitoring/monitor-connection-pool.service.ts:25) reported waiting = 0 in all 2155 samples over 6 h. No request ever queued for a connection, so pool sizing (config.ts:275, poolSize: +(process.env.SQL_POOL_MAX ?? 10)) is not implicated.

3. Client-side amplification

An integration client (Flutter/Dart, identified only by Dart/3.x (dart:io)) fires several POST /v1/auth in parallel for the same not-yet-registered address at startup — measured bursts of 14 requests within 0.5 s and 4 within 0.1 s, with sub-10 ms gaps, i.e. genuinely concurrent rather than sequential retries.

Because auth takes seconds, all of them clear the "does this user exist" check before any commits, then all attempt the insert. The user.address unique constraint rejects the losers with SQLSTATE 23505, which the guard at auth.service.ts:189 catches and converts into a normal sign-in.

That part is working as designed — every one of these requests returns 201 and no user-facing error is produced. It is listed here only because the duplicate-key volume in the database log is a free, zero-cost proxy for the auth tail: it rises when auth gets slower or a client gets more parallel. Over 24 h there were 63 such rejections across 25 distinct addresses, ~80 % of them on Monero addresses, which is consistent with Monero having the widest race window.

Proposed fixes

1. Module-level keep-alive agent for the Monero client, constructed once instead of per call, plus an explicit timeout. Contained change, removes 0.2–1.4 s per Monero sign-in and the unbounded-hang risk. Same treatment for zano-client.ts.
2. Avoid the second verifySignature round trip where the message variant can be determined up front (auth.service.ts:559-561), or at minimum ensure the fallback reuses the pooled socket.
3. Investigate the 95 % event-loop utilization separately. This is the systemic contributor and needs profiling rather than a guess; it inflates every route, not just auth. Worth doing before or alongside 1–2, since it bounds how much 1–2 can actually deliver.
4. Optionally, ask the integrator to serialize or de-duplicate in-flight auth requests per address. Not required for correctness — the 23505 guard handles it — but it removes the burst load at source.

Items 1 and 2 are small and independently shippable. Item 3 is the larger piece.