Skip to content

Run cron jobs and HTTP in separate processes - #4537

Open
TaprootFreak wants to merge 80 commits into
developfrom
perf/cron-process-separation
Open

Run cron jobs and HTTP in separate processes#4537
TaprootFreak wants to merge 80 commits into
developfrom
perf/cron-process-separation

Conversation

@TaprootFreak

@TaprootFreak TaprootFreak commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Important

CRON_ROLE must be set in every environment before this is deployed. The boot aborts
without it — deliberately, see below — so an environment missing the variable fails on the next
deploy, whatever that deploy was for. The two repositories have separate pipelines with no
dependency between them: merging here triggers a deploy without checking whether the variable is
already in place. The configuration change is a separate, already open PR in the infrastructure
repository; it is inert for the currently running image.

Warning

This PR bundles four subsystems, and a revert is all-or-nothing across all four. That is a
deliberate decision, recorded here so it does not surprise anyone at merge time. See
What this PR bundles directly below before approving.

What this PR bundles

The title says process separation, and that is the reason the branch exists. It is not all the
branch contains. Four independently revertable subsystems ended up on one branch, and because they
are one branch they are also one revert: pulling any of them out after the fact pulls out the other
three.

Area What is in here Why it is entangled
Process split CRON_ROLE, CronScope, mandatory scope on 140 @DfxCron declarations, boot log, role heartbeat The subject of the PR
Cross-process lease cron_lease table, migration, entity, CronLeaseService, shutdown handler in main.ts Only meaningful once two processes exist, and the split is what makes the in-process lock insufficient
OpenTelemetry metrics pipeline src/runtime-metrics.ts, new dependencies, new OTEL_METRIC_EXPORT_INTERVAL Written to measure the event loop saturation that motivated the split; independent of it at runtime
MonitoringService rework DB-backed read path, 30 s process cache, pessimistic_write merge on write Forced by the split: with the observers on the worker, the API process has no state of its own to serve
DashboardFinancialService rework Aggregation stays in the worker, response building becomes its own api job, LatestBalanceStore Forced by the split, for the same reason

Two of the five (metrics pipeline, and the lease insofar as it changes the schema) would stand on
their own. They are not being split out: the operator has decided against a follow-up PR, so the
bundling is stated instead of removed. What that means concretely:

  • A revert of this PR removes the OTLP event-loop metrics and any dashboard or alert built on
    them, whether or not the reason for reverting had anything to do with them.
  • A revert re-introduces the monitoring read path that answers from process memory, which is
    wrong in a two-process deployment — so a revert here has to be accompanied by a rollback of the
    role configuration, not just of this code.
  • A revert does not drop the cron_lease table. The migration is not reverted by reverting the
    code; the table is simply left unused.

The safe rollback is not a revert of this PR but CRON_ROLE=all, under which every process
registers every job. That is the mode this branch is designed to be merged in.

Why

Background jobs and HTTP requests share a single Node event loop, and Node runs JavaScript on one
thread. A CPU-heavy job therefore delays every request in the same process.

Measured in production while investigating slow API responses:

Signal Value
Event loop utilization 84% mean, p90 100%
Event loop delay mean 339 ms, p95 up to 5.3 s
/version (no DB access, 1 ms handler), measured locally against the container p50 7 ms, p95 5.5 s, max 16 s
Actual HTTP load 1.9 req/s

A p50 of a few milliseconds next to a p99 in the seconds is the signature of a blocked loop, not of
slow work: the same distribution reproduces on a route touching neither the database nor any
external service, and it persists when the network path is bypassed entirely. Database, host and
network were each ruled out separately — during one 6.1 s freeze the database had zero active
queries, and the connection pool never had a single waiting request.

The cause is the scheduler, not traffic. Grouping the delay measurements by second-within-minute
over six hours (2,154 windows) gives stall rates of 5.8 / 52.8 / 73.3 / 81.8 / 50.0 / 21.2 % — a
factor of 14, following the scatter curve of the job start delay exactly. Request traffic in the
same grid varies by a factor of 2 and peaks elsewhere.

What changes

This PR is the application half of running the same image twice: one process serving HTTP, one
running the background work. Under CRON_ROLE=all no existing @DfxCron is left out on account of
its scope — new are the role heartbeat, the Spark wallet maintenance job, refreshLatestBalance,
the payment delivery job, the WebSocket liveness sweep and the registration logs.

1. CRON_ROLE decides which jobs a process registers, and CronScope says which process a job
belongs to. Three values each, because two would not be enough in either direction:

  • A role all is needed because otherwise no value covers every job — local development, the test
    suite and any deployment without a separate worker would each lose something.
  • A scope api is needed because some work is bound to the process holding the open connections —
    state a request path reads in this process, which a job running elsewhere cannot maintain.
    Delivering to those connections is a different matter and is no longer done from an api-scoped
    job; see The api scope and the lease for
    why that separation had to be made.

The variable has no default and an unknown or empty value aborts the boot. Every possible default
is silent in one direction: worker would make a misconfigured API process run all background work
twice, api would make a misconfigured worker do nothing at all.

2. scope is mandatory on every @DfxCron. A wrong classification fails silently — a job
wrongly scoped worker leaves the cache it maintains empty in the process that reads it, with no
error anywhere. A default plus an exception list moves that decision into a hand-maintained list
the compiler never sees; such a list grows and goes stale, and pinning it in a test proves the
state of the list rather than the property it stands for.

Counted from the source tree rather than carried forward: 140 @DfxCron declarations — 119
worker, 5 api, 16 both
, across 98 files and 34 areas; 118 carry a process flag, 22 do
not, five of those deliberately. 139 of the 140 have a registration path. docs/cron-jobs.md
carries the assignment per job, generated from the decorators, and both of its distribution tables
sum to 140.

3. Cases where a scope alone would not have been enough:

  • Monitoring state (11 observers behind GET /health* and /monitoring/data). Scoping the
    observers to the worker would freeze eleven endpoints at the boot snapshot in the API process,
    the health report included; scoping them to the API process would move AML, node and bank queries
    back into the request path. The state is already persisted in full, so the read path takes it from
    there with a 30-second process cache. The write path needed its own answer: every process writes
    the whole state as a single row, so only the metrics changed in this process are merged into the
    stored row — otherwise whichever writes last drops the other's work.
  • The dashboard balance store, written by the financial aggregation and read without touching
    the database. That property was measured (23 ms median, 1,989 ms p95 before it existed) and is
    kept: building the response becomes its own minute job scoped api, and the aggregation stays in
    the worker.
  • Periodic work registered outside the scheduler — two native @Cron decorators and a bare
    setInterval driving on-chain wallet maintenance. All three would have run in both processes.
    A test forbids the patterns, and carries no exceptions.

4. Four jobs gain a process flag so a single misbehaving job can be stopped at runtime
instead of by deploy, and the worker reports itself under its own service name so its outgoing
calls do not read as API traffic.

Cross-process lease

An architecture review named the load-bearing assumption of the original design: "there is exactly
one worker, and the configuration is right"
. It was held up by convention, a runbook sentence and
an alert that reports a double run 15–25 minutes later. For a path that moves money, detection
is the second-best answer — and LockClass keeps its state in a field in process memory, so it
cannot see a second process at all.

Jobs scoped worker or api hold a lease in the cron_lease table for the duration of their
run.
A single statement decides it: the upsert takes the row over only if it has expired. Two
processes racing are serialised by the primary key and one of them gets a row back.

What that is worth, stated precisely. The lease does not exclude a double run, and it does
not bound how long one lasts. A job runs once because the deployment runs one worker and because
the job tolerates being run again. What the lease adds underneath those two properties is that a
second process has to take the claim before it may start the job: while the holder keeps
renewing, a wrongly configured second process — a missed recreate, a second worker from --scale,
two processes left on all after a rollback — does not start it at all. If the holder stops
renewing while it is still working, the claim lapses, the second process starts, and the first runs
on to its own end; nothing here shortens that, because a running function cannot be aborted from
the outside in JavaScript and a cooperative check at every write is the same work as the fencing
token this does not carry. CronLeaseService says so in its own "What it does not do".

What the lease does bound is the waiting: a claim left behind by a process that was killed
blocks the job for the lease TTL rather than until somebody reads an alert, and that same span is
the longest a second process waits before it may take the job over.

Scope both is deliberately exempt. Those jobs maintain state a request path reads in their
own
process; a lease over them would starve whichever lost the race and freeze that state. Their
safety comes from a different property: running twice has to be harmless by construction, which is
what CONTRIBUTING requires of them.

Why a lease and not an advisory lock. pg_advisory_lock is bound to the connection and would
hold a pooled connection for the whole runtime of the job. 67 jobs declare a timeout measured in
minutes; that is a real risk to a connection pool sized by SQL_POOL_MAX. An expiring row
costs one short query each to take, extend and release.

The lease is 60 seconds, renewed every 20, and unrelated to the job's timeout. The expiry bounds
how long a claim outlives an owner that can no longer speak for itself — SIGKILL, an OOM kill, a
lost machine. That is a property of the failure mode, not of the work, and deriving it from the
job's own timeout got it backwards: timeout is measured in seconds and nineteen @DfxCron
declarations carry 7200, so a process killed mid-run used to block its own successor from that job
for up to two hours, silently.

Shutdown is the other half. Nothing in this repository ever asked for a shutdown hook, so
SIGTERM ended the process instantly and the release in the finally never ran.
CronLeaseService.shutdown now waits up to ten seconds for the runs this process holds, so their
normal release hands the job to the successor. A run still going after that keeps its lease:
taking it away would let the successor start the same job while this process works on it for the
rest of the grace, which is the outcome the lease exists to make rare.

It hangs off a SIGTERM/SIGINT handler rather than app.enableShutdownHooks(), and that is
deliberate. The idiomatic call is global: it would also start running the nine onModuleDestroy
implementations this application carries but has never executed, and Nest runs those before the
lease hook — they empty the strategy registries that PayIn, PayOut and DEX jobs resolve from. On
its own that is harmless, since the process was about to die. Next to this change it is not: the
wait deliberately keeps in-flight jobs alive longer, so a running payout would gain time to fail on
an emptied registry rather than simply be cut off. A test pins that the idiomatic call stays out.

An unusable lease table is reported rather than silent. Without it every worker- and api-scoped
job is skipped on every tick — the right behaviour, and what CONTRIBUTING asks for, but the skip
used to look exactly like a job with nothing to do. The role heartbeat is scope both and
therefore exempt from the lease, so it kept reporting a healthy process while everything it counts
sat out; and it counts registered jobs, which cannot see this at all. The lease layer now reads
the table at start-up and carries a health flag that stays false until an operation gets through;
the heartbeat writes the same line at error level with the reason appended when it is bad. The same
line, because the role alert matches on its shape.

What it still cannot do — and the code says so. If the holder stops renewing while a job runs,
the claim expires and a second process can start the job while the first is still working, for as
long as that first run takes. Full fencing would need a token on every single write. What the lease
shortens is the wait, not the overlap.

With the database unreachable the job does not run. A job that moves money must not proceed on
the assumption that it is probably alone — and that is exactly when the assumption is least safe.

The api scope and the lease pull against each other

PaymentCronService writes to the database and triggers merchant webhooks, which argues for one
process, and it used to be the only thing releasing the callers waiting on the process that ran it,
which argues for every process. A single scope cannot satisfy both, and the lease made that
visible: whichever process lost the claim left its callers waiting for nothing.

The resolution taken is to split the job rather than the scope. The writing runs under the lease
(Worker), and deliverPaymentUpdates delivers from the state those writes leave behind, scoped
Both, in every process, without a lease — it writes nothing and calls nothing outside its own
process, which is what allows it to run everywhere. An api-scoped job losing the race is still
logged at error level, unlike a worker job, which loses it every cycle by design.

Leaving Pending is decided by the update statement rather than by a status read, because the
expiry timers stay in the process that served the request while the expiry job runs in the worker,
and cancelling and completing arrive from request paths as well. Only the caller the database lets
past Pending sends the merchant its webhook.

Schema

migration/1785600000000-AddCronLease.js creates cron_lease. Two things worth stating:

  • The primary key is PK_a12c181c2b26f33be13d55a15afPK_ plus the first 27 characters of
    sha1('cron_lease_name'), which is what TypeORM's own naming strategy produces. A hand-picked
    name would not be recognised by a schema comparison, which would then offer to create the
    constraint. A new test recomputes every primary key declared in a CREATE TABLE across all
    migrations (102 of them, all matching) and rejects any spelled-out constraint name.
  • acquired and expires are timestamptz. They are compared against now() in raw SQL, and a
    value without a zone on one side of that comparison resolves through whatever time zone the
    session carries: the same row expires an hour late or an hour early across a daylight saving
    change. An hour late is a job that runs nowhere, an hour early is two processes running it.

src/shared/models/cron-lease/cron-lease.entity.ts mirrors the table. The service never reads
through a repository — the claim is a single INSERT .. ON CONFLICT .. WHERE the query builder
cannot express — but a table that exists only as DDL is invisible to the entity model, and the next
generated migration would read that absence as an instruction to DROP TABLE "cron_lease". A test
builds the entity metadata without a connection and checks it against the migration file.

Deployment

The order matters and is not optional:

  1. CRON_ROLE=all must be set in every environment before this PR is deployed. The boot aborts
    without it, so an environment missing the variable fails on the next routine deploy — for the
    currently running image the variable is unknown and therefore inert.
  2. Merge and deploy this PR. The migration creating cron_lease runs with it.
  3. Observe: the boot log states the split — CronRole all: registered 139 of 139 jobs (worker: 119, api: 4, both: 16) — and health and dashboard endpoints answer unchanged.
  4. Alerting, log-level normalisation, runbooks and dashboards — before, not after the next step.
  5. Create the second process and set the roles. With the roles split, the boot log reads
    registered 135 of 139 in the worker and registered 20 of 139 in the HTTP process.

Rolling back never means reverting this PR: under CRON_ROLE=all its content is today's behaviour,
so what gets reset is configuration. See What this PR bundles for why a
revert is the expensive option.

Testing

npm test, plus targeted runs on the affected suites; tsc --noEmit, ESLint and Prettier clean.
Coverage added by this branch:

  • every rejected value for the role, including the empty string and the absent variable, asserting
    a throw rather than a silent default
  • the env -> Config wiring the cron service actually reads, not just the parser
  • registration per role: all registers everything, each role drops the other's scope, both
    survives in all three
  • which jobs pass through the lease, and that a job declaring timeout: 7200 still claims for 60
    seconds — the regression that made a deployment block a job for two hours
  • shutdown: the lease survives a shutdown that outlasts the grace period, the shutdown does not
    return before the job does, main.ts is pinned to wire it to the signal at all, and pinned NOT
    to reach for app.enableShutdownHooks()
  • an unusable lease table: reported at start-up, still reported on the next heartbeat when no new
    failure occurred, and reported healthy again once a claim gets through
  • an api-scoped job losing the race is reported; a worker job losing it is not
  • constraint naming across every migration, and the cron_lease entity against its own DDL
  • the monitoring service: read path answers from the persisted state including the filtered
    queries, the merge prefers whichever value is newer, the write path keeps metrics another process
    wrote, the row is read under a write lock in the same transaction it writes in, an older
    measurement is not put back over a newer one, and the merge is retried only on errors a retry can
    resolve
  • the monitoring state row: an environment whose state does not live under id: 1 is answered from
    the row that exists, and the write seeds id: 1 from it rather than with a partial state
  • the statistic start-up fill: not in the worker, yes in api and all, off when the process flag
    is off, and a failure reported instead of left as an unhandled rejection
  • the Spark wallet maintenance: registered as a job rather than a timer, scoped worker
  • the guard against @Cron(, @Interval(, @Timeout( and setInterval(, including a check that
    its exception list still matches something

Every fix in the review rounds below was verified by putting the defect back and watching the test
fail, then restoring it.

Known discrepancies, recorded rather than fixed

ExchangeController::checkTrades is never registered: its class is listed under controllers:
in ExchangeModule and nowhere under providers:, and DiscoveryService.getProviders() does not
return controllers, so the scan never sees the decorator. This predates the process split — the job
has never run. TransactionController::checkLists looks like the same case but is not:
HistoryModule lists that class under both controllers: and providers:, so the job is
registered, on the provider instance, which is a different object from the controller instance the
request handlers use.

CitreaBaseStrategy::checkPayInEntries is declared on an abstract class, so it is registered
once per concrete subclass rather than once per declaration. There is currently one subclass.

16 jobs carry no process flag. That is pre-existing, named in docs/cron-jobs.md as an omission
rather than hidden, and retrofitting it means introducing 16 new kill switches — a decision about
those jobs, not about this PR.

What should happen to any of these is a decision about the jobs, not about this inventory.

Review rounds

Round 1 — rebased onto the current develop, conflict-free, CI 12/12 green.

Round 2 — role heartbeat added. DfxCronService::reportRole writes
CronRole <role>: heartbeat, N jobs registered in every process every ten minutes. The reason lies
outside this repository but the line originates here: the Grafana rule meant to report a wrong role
assignment used to read the boot line, which is written exactly once. On a healthy system a
counting window over it reports permanently from the day after the last deploy, because the line
falls out of the window while the container keeps running. And the most expensive state is
precisely the one without a restart: if the recreate for a configuration change does not happen,
the HTTP process keeps its old role while the worker takes over the same jobs. Without a restart
there is no new boot line, so the rule could not see it structurally. Scope both so the line
appears in every process, with the role in the line; no process flag, because a watchdog that
can be switched off looks, switched off, exactly like the failure it reports. useDelay: false,
because the alert reads a 12-minute window and the jitter is adjustable from outside through
CRON_JOB_DELAY — a watchdog must not have its timing tuned by a knob meant for spreading load.

Round 3 — seven review points, three implemented, four re-measured and declined with the
measurement stated. Implemented: the heartbeat tests were reading decorator metadata and would have
stayed green if the scan never saw the method, so the test now hands the scan its own service
instance the way Nest does and expects the heartbeat in the count; the guard test's setTimeout
gap was documented with its evidence; and the claim "behaviourally identical to today under
CRON_ROLE=all" was narrowed to what is actually true. Declined: the git diff --check whitespace
report (all files are CRLF and .prettierrc sets endOfLine: auto; two consecutive develop
commits report the same), the PaymentCronService scope (see above — the concern was right and is
now addressed by the lease and by reporting the lost race), the 16 missing flags, and the
"environment updates missing" point (they are in the infrastructure repository, because that is
where CRON_ROLE is set).

Round 4 — the cross-process lease, described above.

Round 5 — a five-instance review of this PR and its three infrastructure counterparts. Nine
findings here, all fixed on this branch: the lease TTL derived from the job timeout (up to a
two-hour outage per deployment) and the missing shutdown path; an unusable lease table looking like
a healthy process; a primary key name that violates the deterministic-naming rule; the missing
entity; timestamps without a time zone; the Spark maintenance timer and the statistic start-up fill
both running outside the scheduler and therefore outside the lease; the api scope contradicting
the lease; the monitoring read path depending on a row with id: 1; and this bundling, which is
named here rather than split out.

@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Review rounds 1 and 2 (automated reviewers)

Two rounds so far, four reviewer passes. Both rounds returned no-go and both were right; the findings below were real defects, not style.

Round 1

Correctness pass — no-go. The switch was too coarse: turning off every job on an HTTP-only instance also disabled work that must run there, because it refreshes process-local state that requests on that same instance read.

  • resyncDeniedJwtAddresses / resyncDeniedJwtAccounts rebuild the in-memory JWT denylists. Frozen at boot, blocking an account would no longer revoke its live tokens until the next restart — and the lookup fails open on an empty set, so nothing would have surfaced it.
  • resyncDisabledProcesses applies the database-driven kill switch; updateCache holds the specifications behind quotes and limits.
  • Two native @Cron decorators bypassed the switch entirely, which made the claim that an HTTP-only instance registers nothing false.

Addressed by the perInstance flag: jobs whose effect is confined to their own process keep running everywhere, everything else is skipped. The native decorators moved to @DfxCron.

Conventions pass — go with findings. The empty string was treated as "unset" and would have silently re-enabled the scheduler — the likeliest deployment accident of all (CRON_JOBS_ENABLED=, an unresolved ${VAR}). It now throws. .env.example documents the variable with an explicit value, the tests cover the env -> Config wiring rather than the parser alone, and the sdk-metrics range was aligned with the rest of the OTel stack.

Round 2

Correctness pass — no-go. Round 1 had only reviewed the 21 jobs without a process parameter; the same class exists among those that declare one. Rather than reading the list again, every @DfxCron body was scanned mechanically for writes to process-local state:

  • payment-link-fee.updateFees — cache entries expire after 5 minutes, so LNURL-Pay and payment link quotes would have started failing five minutes after boot on the instance serving all customer traffic.
  • tfa.processCleanupSecretCache — the only thing enforcing expiry of mail 2FA codes and enrollment secrets; verify() never checks the date itself.
  • auth-lnurl.processCleanupAccessToken — likewise the only enforcement of the 30-second window in which a minted JWT is retrievable via the publicly known k1.

The scan also flagged five jobs that merely set a "warning already logged" flag while doing global work (pay-in registration, fiat sync, monitoring observers). Marking those would have processed incoming funds twice — they deliberately stay global.

Verified rather than assumed

  • @Lock(7200)timeout: 7200 is equivalent, and the DisabledProcess guard sits at the same point inside the lock.
  • No @Cron, @Interval, @Timeout, SchedulerRegistry or new CronJob remains outside dfx-cron.service.ts.
  • No import cycle: tracing.ts registers the meter provider at module load, before main.ts imports ./runtime-metrics.
  • Unset OTEL_METRIC_EXPORT_INTERVAL lets the reader default of 60 s apply.
  • npm test: 342 suites, 6227 tests, no regressions.

Known limitation

/health reads the observer state from memory, and the observers write to the database, so they cannot simply run everywhere. On an HTTP-only instance the endpoint reports the snapshot loaded at boot. Making it re-read that snapshot touches observer semantics and belongs in its own change — noted here so it is not discovered in production.

Still open

One reviewer finding is not addressed: spark-client.ts starts a bare setInterval in its constructor that performs an on-chain wallet operation every 5 minutes. It bypasses every scheduler switch and would run in both containers. It predates this PR, but the split makes it duplicate — worth resolving before rollout.

This PR stays in draft. It is not ready for review while that item and the /health limitation stand.

@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Status: the switch is groundwork, not yet safe to use

A full inventory of the codebase (static call graph across all 2066 .ts files, forward BFS from every global cron job and every HTTP entry point, crossed with every process-local state mutation) found that running two instances breaks in eight places, four of them in the money path. Beyond the three already noted in this thread:

  • GET /v1/paymentLink/payment/wait and /v1/lnurlp/wait/:id would hang forever — the resolve comes from a cron job while the wait map lives in the HTTP process, and AsyncMap.wait(id, 0) installs no timer at all, so the promise is never settled either way. Every such request leaks a map entry, a promise and a socket.
  • POS terminals would never receive their device command: the subject fires in the job process, the WebSocket client registry lives in the HTTP process.
  • BuyService.cache has no TTL and its only mutator runs on HTTP, so the job process would never see a newly created buy route — the matching SEPA credit falls through to BankTxType.GSHEET, which is terminal.
  • The EVM nonce cache is per-process and is the only thing tracking sent-but-unmined transactions; both processes send from the same wallet.

What this PR contributes stands on its own and is worth merging: the perInstance classification is the groundwork that inventory builds on, the runtime metrics make event loop saturation measurable without an intervention in production, and the switch itself is inert until an instance actually sets it — unset keeps today's behaviour everywhere.

What must not happen is setting CRON_JOBS_ENABLED=false anywhere before those eight places route their state through the database. The deployment configuration that would do so is deliberately held back.

@TaprootFreak
TaprootFreak force-pushed the perf/cron-process-separation branch from 779a5de to 81a281f Compare August 1, 2026 04:17
@TaprootFreak TaprootFreak changed the title 5c008c4f - Run cron jobs and HTTP in separate instances, and measure event loop saturation Run cron jobs and HTTP in separate processes Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

❌ TypeScript: 2 errors

@TaprootFreak
TaprootFreak force-pushed the perf/cron-process-separation branch from 6b1e74f to bfd4264 Compare August 1, 2026 09:15
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Seven review passes until both reviewers reported no findings, each pass a pair of independent reviews over the full diff — one on conformance and completeness, one on logic and correctness.

The findings that carried weight, and what changed because of them:

The monitoring write could lose a value, in three layers. Merging the changed metrics into the stored row narrowed the race instead of closing it: two writers that both read before either wrote still overwrote each other, and the lost value did not come back, because the next run compares against this process's own previous state and finds the metric unchanged. Read, merge and write now happen in one transaction that locks the row. That surfaced the next layer — a writer that waited on the lock holds a value from before that wait, and wrote it back over a newer one — so a changed metric is only taken over if its timestamp is not older. And an absent row cannot be locked at all, so the merge is retried once, restricted to the errors a retry can resolve.

Two caches were scoped for a process that never fills them. The exchange trade map and the refund list are written and read by request paths of their own controllers; under both the job ran in a process where the map is always empty. Now api, which is what the rule in this PR says.

The enum members did not follow the repository. Measured rather than assumed: 335 enums use ALL_CAPS members against 6 using PascalCase, two of which were the ones added here.

The job inventory gained a scope column. The earlier reasoning — a column would go stale — applies to interval and flag just as much, and those are maintained. All three now come from the decorator arguments.

Also: any removed from a test in favour of the mocking helper the rest of the suite uses, German comments translated, CRLF line endings restored in two files that tooling had converted, the guard extended to @Interval and @Timeout, and the contribution guide's examples made to compile against the mandatory parameter.

Several comments were rewritten rather than softened, because they claimed things a reader cannot verify here: production measurements, how the code came to be, a reference to a dashboard living elsewhere, an assertion about the deployment topology, and a claim about what the provider discovery sees that is simply not true.

Two findings were rejected with reasons: a comment carrying production data is pre-existing code from develop and not part of this diff, and a claim about several API replicas describes a topology that does not exist — the lock lives in the process, which is the property that actually matters and is now what the comment says.

@TaprootFreak
TaprootFreak marked this pull request as ready for review August 1, 2026 09:59
@TaprootFreak
TaprootFreak force-pushed the perf/cron-process-separation branch from 1769bcb to ca85f1f Compare August 1, 2026 11:49
@TaprootFreak

Copy link
Copy Markdown
Collaborator Author

Full audit against CONTRIBUTING.md — two gaps closed, the rest measured clean.

The earlier review passes checked the diff, not the guide. Reading all 1260 lines against these 119 files turned up two things:

1. The guide contradicted itself on bare @Cron. The opening example presented it as the alternative you may pick when you handle @Lock and DisabledProcess yourself — while the section this PR adds states that a test rejects it. Both stood in the same file, three paragraphs apart. A bare @Cron carries no scope and registers in every process, so the guard is the rule; the example only shows what the wrapper takes off your hands. Reworded, with a pointer to the section.

2. refreshLatestBalance runs every minute where the guide asks for fifteen — and the reason was nowhere written down. It is a good one: LogJobService writes the entry this job reads at exactly that interval (TRADING_LOG, EVERY_MINUTE). A longer interval saves no write, it only serves a staler value than the data allows. Now in the doc comment.

Measured clean across the 106 changed non-spec files (added lines only): no : any, no console.log, no TODO/FIXME, no return await, no eslint-disable, no injected dependency that is not private readonly. No entity or column change, so PR-completeness item 1 (migration) does not apply; item 5 (cron inventory) is covered — docs/cron-jobs.md carries all 136 jobs with the new scope column. Branch is off develop; perf/ as a prefix is established practice here (11 branches) even though the guide names only feat/ and fix/. The guard spec sits in __tests__/, which is the convention at 352 files to 10.

@TaprootFreak
TaprootFreak force-pushed the perf/cron-process-separation branch 4 times, most recently from 27adf63 to 45d1744 Compare August 1, 2026 17:23
Background jobs and HTTP requests share a single Node event loop, and Node runs
JavaScript on one thread. A CPU-heavy job therefore delays every request on the
same instance: measured on production, a request to /version - no database
access, handler time 1 ms - took p95 5.5 s when measured locally against the
container, with event loop utilization averaging 84%. The delay followed the
cron jitter window exactly, so it was driven by the scheduler, not by traffic.

CRON_JOBS_ENABLED=false now makes DfxCronService return from onModuleInit
without registering anything, which allows running the jobs in a second
instance while the first only answers HTTP.

The switch is deliberately coarser than DISABLED_PROCESSES: that one only skips
jobs declaring a `process`, and 21 of 131 jobs declare none - among them trade
checks, referral credits and volume resets. Those would keep running on both
instances, and since cron locks are per-process, nothing would catch the
duplicate execution.

An unset variable keeps the previous behaviour so every existing environment is
unaffected. Any value other than 'true'/'false' throws instead of being coerced:
a typo such as 'fals' would otherwise silently re-enable the scheduler on an
instance meant to be HTTP-only, and duplicate execution of financial jobs is far
more damaging than a failed boot.
Traces answer how long a request took, but not why. A request queued behind a
saturated event loop is indistinguishable from one waiting on a slow query, so
diagnosing the recent latency issue required taking a V8 CPU profile against the
running production process - a one-off snapshot obtained through an intervention
in production.

MonitorEventLoopService already measures utilization and loop delay correctly,
but only writes them to the log, and it runs as a cron job: an HTTP-only
instance registers no jobs and would report nothing precisely where saturation
matters most. Collection is therefore driven by the OTel metric reader instead
of the scheduler, so it runs on every instance.

Export reuses the existing OTLP pipeline, so no /metrics endpoint and no
additional scrape target are needed. Instrument names follow the Node.js runtime
semantic conventions. With OTEL_EXPORTER_OTLP_ENDPOINT unset, no meter is
registered and the app boots unchanged, matching how tracing is already handled.
Only a variable that is absent entirely now means "run jobs". An empty value
takes the dangerous path the surrounding comment claims to rule out: an
`CRON_JOBS_ENABLED=` line in an env file or an unresolved `${VAR}` in a compose
file would have re-registered the scheduler on an instance meant to serve HTTP
only, and jobs without a declared `process` would then run on two instances at
once. That accident is more likely than the misspelling the check was written
for, so it fails the boot too.

Also documents the variable in .env.example with an explicit value rather than
the usual empty placeholder, which would now be rejected, and covers the
env -> Config wiring instead of the parser alone: the cron service reads
Config.cronJobsEnabled, and nothing verified that path end to end. The service
test builds its configuration through ConfigService/GetConfig for the same
reason - spreading GetConfig() dropped the prototype getters and left the
global singleton structurally different from a real Configuration.

Shares the telemetry-enabled check between tracing and runtime metrics so the
two halves cannot drift apart, and aligns the sdk-metrics range with the
baseline the rest of the OTel stack pins.
The switch was too coarse. Turning off every job on the HTTP instance also
turned off the jobs that must run precisely there, because they refresh
process-local state that requests on that same instance read:

- resyncDeniedJwtAddresses / resyncDeniedJwtAccounts rebuild the in-memory JWT
  denylists every 30 s. Without them the lists stay frozen at boot, so blocking
  an account would no longer revoke its live tokens on the instance that serves
  HTTP until the next restart. The lookup fails open on an empty set, so nothing
  would have surfaced the gap.
- resyncDisabledProcesses applies the database-driven kill switch and safety
  mode, which gate several HTTP endpoints.
- updateCache holds the transaction specifications behind quote and limit
  calculation; checkLists and processCleanupMailSecretCache expire local caches
  that only grow where the traffic arrives.

DfxCron therefore takes a `perInstance` flag for work whose effect is confined
to its own process, and the switch now skips only the rest. Running such a job
on every instance is harmless by construction; anything that writes to the
database or drives business forward stays global and is skipped as before.

Also moves the two remaining native @Cron decorators to @DfxCron. They were
registered by the Nest scheduler directly and bypassed the switch entirely,
which made the claim that an HTTP-only instance registers nothing false. Their
process guards move into the decorator, where DfxCronService evaluates them
anyway, so the bodies lose their early return.
The first pass only reviewed jobs without a `process` parameter, which missed
the same class of job among those that declare one. Found by scanning every
@DfxCron body for writes to process-local state instead of reading the list
again:

- payment-link-fee.updateFees holds the fee cache behind LNURL-Pay and payment
  link quotes. Entries expire after 5 minutes, so on an HTTP-only instance
  every such request would have started failing five minutes after boot.
- tfa.processCleanupSecretCache is the only thing enforcing expiry of mail 2FA
  codes and enrollment secrets — verify() never checks the expiry date itself.
  Without it those stay valid indefinitely on the instance serving the requests.
- auth-lnurl.processCleanupAccessToken likewise enforces the 30-second window in
  which a minted JWT is retrievable via the publicly known k1; status() does not
  check the timestamp.
- statistic.doUpdate, binance-pay.updateCertificates, monitorEventLoop and
  processCleanupAuthCache follow the same pattern.

The scan also flagged five jobs that merely set a "warning already logged" flag
while doing global work (pay-in registration, fiat sync, monitoring observers).
Those deliberately stay global: marking them would have processed incoming funds
twice.

Known limitation, deliberately not addressed here: /health reads the observer
state from memory, and the observers write to the database, so they cannot
simply run everywhere. On an HTTP-only instance the endpoint therefore reports
the snapshot loaded at boot. Making it re-read that snapshot touches observer
semantics and belongs in its own change.
The Spark client starts a bare setInterval in its constructor that runs
optimizeTokenOutputs against the wallet every five minutes. It predates the
scheduler and bypasses every switch, so with the process split both containers
would drive on-chain maintenance against the same seed.

It is global work and belongs to the job instance, so it now honours the same
flag as the cron jobs. This was the last remaining path by which periodic work
could run outside DfxCronService.
Three files were rewritten with LF by tooling, turning one-line changes into
roughly 1100 lines of diff noise and guaranteed merge conflicts for any parallel
branch touching them. The repository mixes both endings and has no .gitattributes,
so the original ending is restored rather than normalised here.

monitorConnectionPool and monitorConnectionPoolStatic measure this process's own
pool and only log. They are harmless to run everywhere by construction, and
without them pool visibility would be lost on exactly the instance whose pool
serves the requests - the same reasoning that put perInstance on monitorEventLoop.
Every new cron job now carries a classification decision, and getting it wrong
fails silently on the HTTP instance — the JWT denylists were the first example,
three more surfaced in later review rounds. The rule therefore belongs in the
cron section of CONTRIBUTING, not only in the TSDoc of the flag.

Also corrects a comment in runtime-metrics.ts that no longer held: monitorEventLoop
became per-instance in the meantime, so it does run on an HTTP-only instance. The
reason for the metric stands - a queryable series instead of a log line, and
independent of how the scheduler is configured.
CRON_JOBS_ENABLED could express "this process runs the jobs" and "this process
runs none", but not "this job belongs to the process serving HTTP". That third
case exists: a job maintaining state that only a request path reads, or driving
work bound to the connections a process holds open, is worse off in the worker
than it is today. The flag also carried its answer in the wrong place - an
optional perInstance defaulting to "global" points the silent failure at the
dangerous side, because a job wrongly left unmarked stops refreshing the cache
that requests on the HTTP process read, without an error anywhere.

CRON_ROLE replaces it with three operating modes and CronScope with three job
properties. A role runs its own scope plus `both`; `all` runs everything and is
the single-process mode, which keeps local development, the test suite and every
environment without a separate worker exactly as they are.

The variable has no default and an unknown or empty value aborts the boot. Every
default is silent in one direction: defaulting to `worker` makes a misconfigured
API process run all background work a second time, defaulting to `api` makes a
misconfigured worker do nothing at all. Neither raises an error, and duplicate
execution of financial jobs is worse than a failed boot.

The existing perInstance markings move to scope: `both` for the fifteen jobs
refreshing state that both sides read, with two corrections. StatisticService
becomes `api` - the hourly aggregation only feeds GET /statistic, so running it
in the worker would compute a value nobody there retrieves. BinancePayService
becomes `worker` - getCertificates() refreshes lazily on read, so its cache
cannot go stale in a process that never runs the job.

Registration also logs one line per process stating which split actually applies.
A job reaching the scheduler through a dynamically resolved provider or an
abstract base class is counted there and nowhere else, so the effective
assignment is read from the log rather than inferred from a list.

The Spark token optimization follows the same rule from its own timer: it is
on-chain wallet maintenance and must run in exactly one process.
The role only decides which jobs a process registers; it cannot decide which
process a job belongs to. That answer has to come from the job, and leaving it
optional puts the silent failure on the wrong side: a job that forgets the field
keeps running everywhere, and one wrongly scoped `worker` stops refreshing the
cache the request path reads, in both cases without an error.

The alternative - a default plus a list of exceptions - moves the decision into
a hand-maintained list the compiler never sees. Such a list grows, goes stale,
and pinning it in a test proves the state of the list rather than the property it
stands for. Making the field mandatory moves the check into the compiler, where
it stays complete for every job added from here on.

The price is paid once, here: 116 decorators gain `scope: CronScope.Worker`,
which is what they already did and what any job touching only the database or an
external system should do. The seventeen exceptions were classified in the
previous commit. The change is mechanical and carries no behaviour: with
CRON_ROLE=all every job is registered regardless of its scope.
The runtime kill switch works per Process, and a job declaring none cannot be
stopped without a deploy. That is the tool the split relies on when a single job
misbehaves in the process it was moved to, so the three jobs where stopping one
matters get a flag:

- DexService::finalizePurchaseOrders writes liquidity orders, the only one of the
  three carrying real financial risk.
- RefService::checkRefs removes rows.
- JwtRevocationSyncService::syncDeniedJwtAccounts writes a setting. It is
  idempotent, but it should run once rather than not at all.

The remaining jobs without a flag are the monthly and annual volume resets. Their
statement finds nothing left to do on a second run, so a flag would be good
practice but is not a precondition for anything here.
Two native @Cron decorators and one bare setInterval had grown alongside
DfxCronService and are invisible to it. A scope cannot reach them, so after the
split they would run in both processes: the transaction request sync deletes
rows, and the Spark timer performs on-chain wallet maintenance against a single
seed. Nothing in the repository would have flagged the next one.

Both decorators had already moved to @DfxCron; txRequestStatusSync now also
carries useDelay: false. DfxCron staggers job starts by default, and for a
minute-based expression that spreads them over up to 30 seconds - without the
flag the move would have changed when the job runs, not just how it is
registered. The daily job reaches no branch of the delay logic and is unaffected.

The guard test is deliberately syntactic: it asks whether @Cron( or setInterval(
occurs, not whether the code behind it is safe. Its exception list therefore has
a natural ceiling, and the one entry - a timer tied to the lifetime of a client
object rather than to a schedule - is a decision, not a special case. A third
test asserts each exception still matches something, so a leftover entry cannot
quietly read as a rule that still applies.
The observers maintain their results in a BehaviorSubject inside the process
running them, and eleven endpoints read that field: GET /monitoring/data and the
six health paths. Scoping the observers to the worker would leave those answers
frozen at the boot snapshot in the API process - the health report included -
while scoping them to the API process would move AML, node and bank queries back
into the request path, which is the work the split exists to move out.

Neither is necessary, because the state is already persisted in full and reloaded
at boot. Only the read path did not use it. It does now, with a 30-second process
cache in front and one shared read for concurrent requests. The refresh sits
before the branching: /monitoring/data takes subsystem and metric as query
parameters, and refreshing only the unfiltered branch would leave exactly the
filtered answers stale. What this process holds more recently still wins, so a
single-process setup is as current as before and a value arriving through the
webhook is visible before the next write.

The write path needed its own answer. Every process subscribes to its own updates
and writes the whole state as a single row, so whichever writes last would drop
the other's work entirely - the API process overwriting the observers with its
boot state, or the webhook value being overwritten by the next observer run. Only
the metrics changed in this process are now merged into the stored row. Binding
the write to one role would have been smaller, but it would leave the webhook
path silently ineffective rather than working.

The reload does not send mail on failure. At boot that notification runs once; on
a path taken by every request it would answer a database outage with a flood of
mail, during the outage.
processExpiredPayments and checkTxConfirmations both reach
PaymentLinkPaymentService.doSave(), which resolves the AsyncMap that
PaymentLinkController's waitForPayment holds open and pushes the device
activation into the RxJS subject the gateway delivers to its connected clients.
Both are process-local, and both are read only from an HTTP path.

The worker has no ingress, so no point-of-sale device ever connects to it. Run
there, the confirmation would reach nobody: the waiting request hangs until the
connection drops, because waitForPayment passes timeout 0 and AsyncMap.wait
creates no timer for that, so the promise is never rejected.

`Both` is no option: the jobs write to the database and trigger merchant
webhooks, which two processes without a shared lock would do twice.

The cost is deliberate - the confirmation work, blockchain client calls included,
stays in the request process. Moving it would mean waking a waiting connection
across process boundaries, and introducing that into the live payment path is a
larger risk than leaving the work where it already runs. This is also why a
binary cron switch could not express the split: some work is bound to the process
holding the open connections.
waitForPayment waited without an upper bound. A client that hangs up says
nothing the server can hear, so its entry stayed in the wait map and in
waitStates for the lifetime of the process — that is where the growth came
from, not from a side effect of it. The wait is now bounded and the waiter
clears its own entry on the way out, whichever way it leaves. The endpoints
keep their shape: when the bound elapses they answer with the payment as it
stands.

The device register is no longer mirrored into the service. The gateway owns
the sockets, so it is asked what is connected instead of reporting connects and
disconnects into a second map that could drift from them — which is what the
reference count did when one close path fired twice. Registration is bound to
the socket and to every way it can end, including the error path, and a
ping/pong sweep drops sockets that stopped answering, since a peer that
disappears without closing fires no event at all.

The delivery window is taken per device rather than as one minimum across all
of them, where the quietest device set the window for everyone.
CronLeaseService already stated under 'What it does not do' that a lease can
expire under a running job and let a second process start it. Twelve other
places said the opposite — 'at most one process', 'closes that by
construction', 'makes it structural', 'the double run the lease exists to
prevent'. On a money path that difference decides what an operator does during
an incident: told the double run is impossible, running a second worker looks
safe.

They now all say the same thing. The window is bounded by the lease TTL instead
of running until someone reads an alert; what a job scoped worker rests on is a
deployment that runs one worker and a job that tolerates a repeat; the lease is
defence in depth over those two rather than a replacement for either.
Losing the lease neither stops nor pauses the run that lost it, so the
overlap ends when that run ends - up to the two hours the longest job
timeout allows. What is bounded is how long a claim left behind by a dead
process blocks the job, and how long a second process waits before it may
take the job over.

Also drops a stop grace period taken from the deployment configuration,
an empty test exception list the documentation described as populated,
and a heartbeat selector anchored at the one end of the line that carries
caller-supplied text.
processExpiredPayments runs in the worker while the expiry timers stay in
the process that served the request, and cancelling and completing arrive
from request paths, so several processes can read the same payment as
Pending and each send the merchant its webhook and cancel the quotes
again.

A conditional update on the status decides who performs the transition;
only the affected row triggers what follows. It holds for any number of
processes and for every path into the transition.
The start-up call bypasses the scheduler, so it applied the role but not
Process.LATEST_BALANCE_CACHE. Switching the job off still left one run
per deployment.
The connection sweep drops its entry before terminating the socket, so a
terminate that throws cannot leave the entry behind and stall every
following sweep on it.

The comments: the payment-link domain has three Worker jobs, not one; the
metric export interval is only read when an OTLP endpoint is configured;
and the scope on ExchangeController::checkTrades applies to a job that is
not registered anywhere, because the class is a controller.
The conditional update committed on its own, so a caller that stopped
between it and the quote cancellations left a payment out of Pending
with its quotes still open. processExpiredPayments asks for Pending and
would never see that row again.

The effects that write to the database now run in the transaction that
takes the transition, and the merchant webhook stays outside it.
npm resolved the caret range to a version above the one the six
exporter packages depend on, so two copies of @opentelemetry/sdk-metrics
sat in the tree: the meter provider was built against one and read
through the other. The lockfile now holds a single copy.
An editing step rewrote both files with LF, which showed up as a full
rewrite of files that carry three changed lines each. Also drops a local
scratch file that never belonged in the tree.
The store said there was one writer per process, but the job filling it
is leased: with more than one API process, one takes the tick and the
rest never run it, so their store stayed at whatever they started with.
It now loads on demand through AsyncCache and ages out after a minute,
which is what CONTRIBUTING asks of a cache a request path reads; the job
is refresh only, and the start-up fill it replaced is gone.
The comment stated how many declarations carry the timeout and called it
the longest value in the repository. No test holds either claim, and the
next job added makes both wrong without a word.
The comment claimed the activations were closed before the transition,
which holds only when the quote reached a state PaymentQuoteFinalStates
lists - a completion threshold of TX_RECEIVED does not.
The lease decides who runs the job, not how long two runs overlap; in the shutdown case that is the container's stop grace period.
AsyncCache hands a running update to everyone who asks for the entry while it is in flight, which is what keeps a restart from starting one aggregation per request. The read relies on it and does not implement it, so it is asserted rather than assumed.
The device delivery kept a per-device high-water mark and moved it to the
newest `updated` it had read. A payment is stamped by the statement that
writes it and becomes readable only when that write commits, so a mark past
the stamp of a row still in flight asks for `updated > since` and never sees
that row again. Two rows stamped alike are the plainest case; any write
outlived by the read beside it does the same.

A span measured against the present cannot skip a row that way, and it bounds
the read for a connection of any age — the mark only did so while payments
kept arriving. What the span admits twice, a record of the states already sent
answers for, per payment rather than one slot per device, and that record is
dropped with the payments that leave the window.
The transaction tests counted the effects but never checked what they ran on,
so removing the manager from the calls left them green — and the manager is
the whole property: an effect on any other one is a statement that commits
whether the transition does or not.
The read that tells a connected device its payment is through was bounded by
`updated`, a column every write stamps. Two rounds of review found the same
failure through it twice: a mark advanced past a row that had not committed,
and then a span against the present that a transaction outliving it walks a row
straight past. Both end the same way — the device is never told, and nothing
can bring the row back into the read.

`expiryDate` is given at insert and never moved afterwards, so a late commit
can only make a row appear later, never make it skip. The read now selects on
it, one cutoff for every connected device, reaching back past the payment's own
end AND past the configured delay before an expiry is acted on — read from the
configuration rather than assumed, so raising it cannot silently drop the
expiry transition out of the read.

The connection time goes with it. What a device is owed follows from the
payments, not from when it happened to connect, so a device that reconnects is
owed what it was owed before: the delivery record now ages out on the same
cutoff the query uses instead of being dropped with the connection, and a
device whose entries have all gone leaves with them.
Four summaries described the lease as held "for the duration of its run" or
"for as long as the job runs", and one described the renewal as keeping the
claim alive. Each is the short form of a passage that goes on to take it back:
the claim can lapse while the job is still working, and losing it stops
nothing. A reader who takes only the short form takes away a stronger property
than the design has.

Shortened rather than qualified, because this is the fourth round in which the
same class of sentence has come back. What the lease does is claimed before the
start; what happens after that is in "What it does not do", where it already
was.
Round seven found two places where a boundary drawn in an earlier round stops
one statement short of where the damage is.

In `handleQuoteChange`, the activations of a final quote were closed BEFORE the
status moved. If the transition then failed, the payment stayed `Pending` with
its activations closed for good — and nothing brings that back: the quote is
already final, so `checkTxConfirmations` does not return to it, while
`processExpiredPayments` only ever asks for `Pending` and would expire a
payment that was in fact paid. The close now travels with the transition on the
path that has one, and stays where it was on the paths that do not.

In `stateRow`, two writers converging an old snapshot row onto `id: 1` could
lose the first one's work. Both miss `id: 1`, both queue for the old row, and
the second wakes up after the first has created `id: 1` and committed —
merging from the old row then writes a state that predates it. Since this path
only writes what changed, the lost metric does not come back on its own. The
lookup now asks a second time after the wait.

Two statements were also promising more than they hold. The delivery grace said
a late commit can never make a row skip; that is true of WRITES, and the read
still ends somewhere — a transition after that end is missed like any other.
The span now reaches an hour instead of ten minutes, chosen against what
actually delays the transition: the worker being gone, which the silence alert
only reports after seventeen. And the shutdown release is best effort, not a
guarantee: a claim still being taken is not in `inFlight`, so an exit in
between leaves the row to lapse on its TTL — the bound that always applies.

Each of the four is pinned by a test whose counter-proof was seen red.
Round eight, seven findings, none of them in the lease or the role assignment.

The delivery job carried a `process` flag like any other job. It is not a job in
that sense: it is the bridge that carries a result from the process that wrote
it to the process holding the connection. Off in the single-process setup
nothing happens, because `doSave` delivers directly there — so the flag looks
harmless right up to the moment it is not. After the split, switching it off
silently cuts delivery to everything attached to the other container, and no
alert sees it: every process still reports its role and a usable lease. The
flag is gone, and with it the enum value this branch had added for it. The same
reasoning already governs the role heartbeat and `checkConnections`.

The delivery record is written BEFORE the send, and that ordering is now stated
rather than implied — together with what it costs and why the alternative costs
more. The gateway no longer leaves a socket in the map that threw on `send`: a
device unreachable through it would otherwise keep being selected, and counted
as delivered. Two comments promised a reconnecting device everything it was
owed; they now say "within the span the read covers".

Three more: the heartbeat's error form was abbreviated in the comment that
declares it an interface, while the alert reads the full line. Two comments
still said the alert reports a double run — it reports a wrong ROLE, which is
what the logs can distinguish. And the spark job caught its own error to log it
again, which is the redundant try-catch CONTRIBUTING names by example.

The transaction proof had a hole the review was right to name: the existing
tests show the effect services are HANDED the manager, which a service that
ignores it would also pass. The new spec asserts the other direction — given a
manager the write goes through that manager's repository and never through the
injected one, and without one it goes through the injected one.
Round nine found the delivery could lose a command silently, and the code said
the opposite in two places that could not both be true.

The record of what a device has been told is keyed by DEVICE and outlives the
connection — `pruneDeliveries` keeps it that way on purpose, so a reconnecting
device is not told twice. The write path meanwhile claimed that a failed send
would go out again "on the next tick under a new connection". It would not: the
record already said delivered, and the periodic delivery that would repair it
takes the early return. A `send` that threw was therefore lost for good, on the
one path built to prevent exactly that.

The gateway now hands the command over through a sink that answers whether it
reached a socket, and the record is written only on `true`. That is what the
RxJS subject could not do: a subject carries a value one way and swallows what
the subscriber does with it, including a throw. It had one consumer, so it is
gone rather than wrapped.

The cost of this order is a repeat if the process dies between the send and the
record. Repeating is what `waitState` makes harmless; losing is not.

Also: `docs/cron-jobs.md` counted 118 of 140 jobs as flagged with six
deliberate exceptions. Removing the delivery job's flag last round made that
117, 23 and seven — recounted from the table rather than adjusted by one, and
the delivery job now appears in the list of deliberate exceptions with its
reason instead of silently among the omissions.
Round nine moved the delivery record behind the send so a failed one would be
retried. Round ten found the half of that which `send` cannot report.

`ws.send()` throws only while the socket is still CONNECTING. On one that is
CLOSING or CLOSED it takes the call without a word and raises the failure
asynchronously through `'error'` — by which time the sink has already answered
`true` and the delivery has recorded the command against a state it will not
send again, not even when the device reconnects. That is the same silent loss
as before, reached through the one door the try/catch does not cover.

So the state is read first and only an open socket is written to; one that is
not open is dropped, exactly like one that throws. What remains is a socket
closing BETWEEN the check and the send — one synchronous call wide, and the
delivery survives that the way it survives a restart, because the payment stays
inside the read until its own end. What it does not survive is being told
`true` for a socket that was already gone.

Three smaller ones. Two comments still described the RxJS subject that round
nine replaced — a later reader would have looked for a subscription instead of
the sink and missed that it answers. `PaymentLinkFeeService` justified its
`Api` scope with "one of the Api-scoped payment crons"; it is the only one in
the domain, and it writes the cache rather than reading it. And two imports
were not alphabetical.
The cross-repo review found a blocker in the rollout, not in a single PR — the
kind only that review can see.

Between the production release of this application version (step 5) and the
release that brings the lease alert (step 9), production runs with the lease in
force and nothing watching it. `guardAcrossProcesses` does not look at the
role, so under `CRON_ROLE=all` all 123 non-`BOTH` jobs go through the claim. An
unreachable table — a missing grant, a migration that did not run, a database
hiccup — made every one of them skip, payouts included, while every deployed
rule stayed green.

Under `all` the deployment runs ONE process. That is the shape the API had
before this branch existed, with no lease at all. Skipping there made the lease
strictly worse than its own absence, which cannot be the right answer for a
mechanism that is supposed to add safety. So under `all` the job now runs, and
the failure is logged loudly and carried out in the heartbeat. Under `api` or
`worker` the skip stands: there the lease is the only separation, and running
anyway is the double run it exists to prevent.

Four more from the same round.

The heartbeat is now written once at boot. It fires on fixed marks without
jitter, so a process that comes up at :11 and misses :10 wrote nothing until
:20 — against a twelve-minute window, an ordinary deploy could produce the
critical "worker is silent" alarm.

The gateway drops what it believes it told a device when a socket reports
`'error'`. That is how `ws` reports a send on a socket that closed between the
state check and the call, and the comment that dismissed it was wrong: it
claimed the record would age out and the delivery retry. It cannot — the record
ages on the same cutoff the query uses, so "record gone" and "payment still in
the read" exclude each other.

`PaymentLinkFeeService.getMinFee` loads on demand instead of answering
`undefined`. CONTRIBUTING requires that of every cache read in a request path,
and this PR made it bite: the refresh is now `Api`-scoped and leased, so among
several API processes only one wins it per tick.

And `SET LOCAL lock_timeout` is scoped to the whole migration batch, not to the
statement below it — the comment said the first and argued the second.
Round twelve found that the previous round's fix left two holes and eleven
sentences describing the behaviour it replaced.

The fail-open path called the task directly. That skipped the two things the
healthy path does between the claim and the call, and neither is about the
lease: the run never entered `inFlight`, so `shutdown` gave it no grace period
and reported nothing was running; and it started even if shutdown had begun
during the failed claim — a WIDER window than the healthy one, because a
failing attempt runs to the database timeout. A payout could be cut off
part-way with the shutdown log saying the process was idle.

Both paths now go through `track`, which owns exactly those two: the late
shutdown check and the `inFlight` entry. What the claim-holding path adds on
top — stop renewing, hand the claim back — it passes in.

Eleven statements still described the lease as fail-closed everywhere. Five in
this repo: the boot error line a reader sees in production, the `run` doc
twelve lines above the code doing the opposite, `takeFailures`, `reportRole`
and `onModuleInit`. They now say what depends on the role and what does not —
under `api`/`worker` the jobs stop, under `all` they run without a claim, and
both are states to fix, which is why the heartbeat carries either out.

The boot line branches too: under `all` it would have told an operator the
opposite of what the process does.
Three things the on-demand load in getMinFee left open, all of them consequences
of the same move — a value that used to be produced by one process for itself is
now produced in one process and read in another.

onModuleInit ran the full sweep in EVERY process. It is a Nest lifecycle hook,
not a cron job, so the `Api` scope on updateFees never reached it: at boot the
worker queried gas prices for eight EVM chains plus Bitcoin and Firo estimates to
fill a map nothing in that process reads — exactly what the scope exists to
prevent. The hook now asks whether this process serves requests, which is the
property that makes a warm cache worth anything; the role-to-scope table stays in
one place, in runsInThisRole.

The load carried no LOC guard, while the job it complements does. Locally there
are no node connections to ask, so the cache stays empty by design and every
request would have run into the timeout of every one of those calls. It answers
undefined again, as it did before this method loaded anything.

And it started one fetch per caller: the cache is written when a load RESOLVES,
so a burst of quotes for the same chain all saw the same empty entry. That is not
hypothetical — updateFees is leased, so among several API processes only one wins
it per tick and the others rely on this path for a whole minute. Concurrent
callers now share the load in flight; a failed one is dropped in `finally`, so
the next caller retries instead of inheriting it.

Proven by 12 tests, each of the three guards checked against its own removal.
@TaprootFreak
TaprootFreak force-pushed the perf/cron-process-separation branch from c026a1b to 09fe4f6 Compare August 2, 2026 20:25
The comment explained why waiting for the record to age out does not repair a
socket that closed between the state check and the send — and framed it as a
correction of a previous version of itself. The reason is the same either way,
and the reader of a merged file has no previous version to compare against.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant