fix(lightning): survive bad LND WebSocket frames and keep the streams reconnecting

[Danswar]

Problem

The API process died with an uncaught exception:

TypeError: Cannot read properties of undefined (reading 'logger')
    at mapPaymentMessage (dist/integration/blockchain/lightning/services/lightning-ws.service.js:105:14)
    at rxjs/internal/operators/map.js:10:37
    at WebSocket.<anonymous> (dist/integration/blockchain/lightning/lightning-ws-client.js:62:22)
→ reportUnhandledError → throw err → process exit

lightning-ws.service.ts piped all three LND streams through map(this.mapXMessage), passing the
mappers unbound — RxJS calls them with this === undefined.

The happy path never dereferences this, which is why this survived unnoticed. The result-missing
path does, on the very line meant to report the problem:

this.logger.error(`Result not available in payment message: ${paymentMessage}`);

LND's REST gateway terminates a stream with an error frame that carries no result, so a cancelled
TrackPayments subscription was enough to hit it. The process exit tore down all three WebSocket
connections; LND logged the matching TrackPayments: context canceled, close 1006 and
write to closed pipe errors on its side. The container restarted ~4 s later and reconnected.

The same latent bug sat in mapOnchainMessage and mapInvoiceMessage.

Why the existing try/catch did not help

lightning-ws-client.ts wraps this.next(JSON.parse(message)) in a try/catch, which looks like it
covers this. It does not: RxJS catches a throw inside a map projection and converts it into an
error notification on the subject, so it never propagates back to that catch. The subscribers in
lightning-transaction.service.ts register no error handler, so the notification ends up in
reportUnhandledError, which rethrows on a timer and kills the process.

Binding alone is therefore not sufficient — any future throw in a projection would terminate the
subscription permanently, leaving the socket connected while nothing gets processed.

The part that is easy to miss

Removing the crash also removes an accidental safety net. The process dying and restarting was
what rebuilt dead WebSocket connections. Once the process survives, every path that gives up
reconnecting becomes permanent, and every one of them was silent. Most of this PR is about that.

Changes

Mapping (lightning-ws.service.ts)

• bind the three mappers via arrow functions
• route them through safeMap, so a mapping failure logs and yields undefined instead of
  terminating the stream. undefined is already the established "skip this message" signal — the
  handlers in lightning-transaction.service.ts all guard with if (message)
• safeMap takes a zero-argument mapper, so a bare this.mapXMessage reference is a compile error
  rather than a convention
• log the offending frame via describe(), which is total: JSON.stringify rejects circular
  structures and String() throws on a null-prototype object, and this runs inside the handler
  whose whole job is to keep failures off the stream. The old template literal rendered
  [object Object], which is why the incident left no record of what LND actually sent

Reconnection (lightning-ws-client.ts)

• retries no longer stop. The old code gave up after 30 attempts; that was survivable only
  because the process died and came back. An LND outage longer than five minutes would now end
  ingestion for the lifetime of the process
• two health signals. How often the stream reconnects catches one that connects and drops
  straight back — downtime stays short, but LND sends no backlog on subscribe so every gap loses
  events. How long it has been down reports the size of an outage rather than just its existence,
  reaches a stream that cannot connect about 2.4x sooner (300 s against ~725 s), and does not depend
  on handshakeTimeout continuing to keep each attempt short enough to register as a rate. (A rate
  alone once had a real blind spot — a cycle longer than ~124 s never reaches 30 in an hour, and
  before handshakeTimeout a dropped port sat in exactly that band at ~137 s. That band is closed
  now; the downtime signal earns its place on the three reasons above.)
• handshakeTimeout. A stalled upgrade emitted neither error nor close, so the retry chain
  had nothing to react to and the stream stopped for good without reconnecting. Measured against a
  blackholed SYN with the shipped 15 s value: error + close at 15021 ms with the option, no
  event at all for 30 s without it
• every error log on the retry path is bounded. Retrying forever turned the error handler
  and the construction-failure log into one error-level trace per attempt, forever, for an
  unreachable LND. Each outage now reports each distinct cause once, capped — so a repeat stays
  quiet, a cause that changes mid-outage (a refused connection becoming a rejected macaroon) is
  still reported, an LND crash-loop alternating two causes cannot push it back to one line per
  attempt, and a set that saturates says so once rather than letting whichever causes arrived first
  read as the complete story
• new error logs: no connection for N sec. (at most once per 300 s, N being the observed
  outage), N reconnects within M sec. (at most once per 30 reconnects, M being the observed span),
  and further <connection error|retry failure> causes suppressed (once per cause set per reset,
  and only after 5 distinct causes have already been logged). This is new alert surface — a healthy
  stream reconnects only on an LND restart or a deploy and stays silent
• the reconnect timer catches and re-arms. createWebSocket() reads the TLS certificate per
  instantiation and throws on a missing file, so a cert rotation during a reconnect was an uncaught
  exit — and with no socket created, no close event would ever drive the next attempt
• handlers bind the socket they were registered on instead of this.webSocket, so a retired socket
  cannot drive its successor. ws throws synchronously from send()/pong() while a socket is
  still CONNECTING, so this was a live process-exit path of the same aliasing class as the
  original bug
• the message handler is typed WebSocket.Data and parses via toString(); ws delivers a Buffer.
  Behaviour is unchanged — JSON.parse already coerced it — this is a type honesty fix
• the helper that derives a cause key is total, and keys on err.code as well as err.message.
  String() throws on a null-prototype object, and it runs inside the retry catch, where a throw
  skips the reschedule and escapes the timer — a dead stream plus an uncaught exception. Separately,
  from Node 20 autoSelectFamily is on by default and a connect failure to a hostname arrives as an
  AggregateError with an empty message, so without the code every errno collapsed onto one key.
  Latent on the pinned Node 18 image, live on the next bump

Tests

Two new specs. Measured per test rather than asserted:

baseline	failing tests
develop	32
previous commit	2

Covered: every mapper's result-missing path, an unmappable frame, a circular frame, a frame that
neither JSON.stringify nor String() can render, a null frame on each of the three streams
(the one input that makes every mapper throw, and so the only one that tells a guarded pipe from an
unguarded one), stream survival afterwards, invoice mapping
including both settle_date branches, reconnect-after-close, retry past the alert threshold, retry
when socket construction throws, retired-socket isolation, and the three reconnect-rate shapes
(cannot connect however slowly it fails / constant reconnects despite good cycles / healthy stream
stays silent), each alert's throttle and reported figure, the monotonic clock, the subscribe body
being sent on the socket that opened, and — for the two per-outage cause sets — their distinct reset
points, the cap in both directions, an alternating cause, two errors differing only by code, an
error carrying neither, an error that cannot be described at all, and the notice emitted once when a
set saturates.

Every branch is mutation-checked: removing safeMap from any of the three pipes, negating either
signal, dropping either alert throttle or its re-arm, either cause set's dedup, cap, reset, notice
or its label, the handshake timeout, the monotonic clock, either socket binding, the subscribe send,
or the cause key's code or message component each break at least one test. The survivors left in
place are equivalent for every input ws or Node can produce — argument order and separator within
the cause key, and the empty-string filter, which changes the key text but not the partition.

Simulated over 24 h, except the healthy row, which runs a week. The first row is the shape that motivated the downtime signal — a dropped port
used to cycle every ~137 s (a ~127 s TCP SYN timeout plus the 10 s retry wait), under any
reconnect-rate threshold;
handshakeTimeout now bounds it, and the signal covers it either way:

shape	on develop	now
dropped port, slow-failing connect	gave up after 30 reconnects, then permanently dead	reported by both signals, keeps retrying
refused port, 10 s cycle	gave up after 30 reconnects, then permanently dead	reported by both signals, keeps retrying
flap, 28 short cycles + 1 long	gave up after 30 reconnects, then permanently dead	19 reconnect alerts, keeps retrying
healthy, reconnect every 2 h for a week	silent	silent

The develop column is measured against that branch, not against an intermediate state of this one:
its retry budget resets only when a message arrives, so any stream that reconnects without
delivering anything drains it and stops for good. That is the behaviour the process crash used to
paper over.

Local run of the PR CI steps: npm run build, npm run lint, npm run test (78/78) all pass.

Deliberately not in this PR

• The six subscribe() call sites register no error handler. Adding a log-only handler would
  convert a crash-and-restart, which recovers, into a silently dead subscription, since an error
  notification terminates it permanently. safeMap removes the known producer; the rest wants a
  resubscribe strategy, not a swallow.
• alchemy-webhook.service.ts:51 has the same unguarded-projection shape:
  .pipe(filter((data) => network === Network[data.event.network]));. Different subsystem, and it
  sits behind an HMAC check.
• mapPaymentMessage on a truthy-but-empty result emits a NaN-filled DTO. It is contained
  downstream by an existing if (!state) throw and its behaviour is unchanged by this PR.
• Half-open connection detection. A connection whose peer vanishes without a FIN produces no
  traffic and no close, so both signals above read it as a healthy quiet stream. The obvious fix is
  a ping heartbeat, but lnd's WebSocket proxy installs no ping handler — a client ping is answered
  only by gorilla's default, which writes the pong with a 1 s deadline and swallows a timeout — so
  terminating on a missed pong risks reconnecting all three streams on a healthy node. This needs
  one observation against a live LND (does an unsolicited client ping come back?) before it can be
  built safely.
• Missed events during a reconnect. The invoice and on-chain streams resubscribe from index /
  height 0 and replay; TrackPayments does not, so a payment reaching a terminal state during a
  gap is not re-emitted. The daily sync jobs backfill, but the gap is real.

[Danswar]

Nine review passes were needed to get this to zero findings, which is a lot for a fix that started
as one unbound method reference. Worth saying why, since it shapes what is worth looking at hardest.

Removing the crash removed an accidental safety net: the process dying and restarting was what
rebuilt dead WebSocket connections. Once the process survives, every path that stops reconnecting
becomes permanent, and every one of them was silent. Most of the review went on that, and the same
mistake recurred in new shapes — a give-up branch, then a health signal with a blind spot, then its
replacement with a different blind spot, then a log throttle that hid the cause that mattered.

Things that were found and fixed rather than shipped:

• the retry budget drained monotonically for a stream that reconnects but stays quiet, so surviving
  a bad frame traded a loud crash for a silent permanent stop
• a reconnect-rate signal alone has a floor: any repeating failure cycle over ~124 s never registers,
  and a dropped port sat exactly there before handshakeTimeout
• a ping heartbeat was added and then removed — lnd's WebSocket proxy installs no ping handler and
  gorilla's default pong write has a 1 s deadline whose timeout is swallowed, so terminating on a
  missed pong could have reconnected all three streams on a healthy node
• the mapping guard was proved on one stream of three; a null frame is what tells a guarded pipe
  from an unguarded one

Reviewers to point at the sharpest parts: the two health signals and their thresholds
(lightning-ws-client.ts), and whether the new error-level lines are the right alert surface.

Deliberately out of scope, listed in the description: half-open connection detection (needs one
observation against a live LND first), the six subscribe() sites without error handlers, and the
same unguarded-projection shape in the Alchemy webhook service.