Run cron jobs and HTTP in separate processes - #4537
Conversation
Review rounds 1 and 2 (automated reviewers)Two rounds so far, four reviewer passes. Both rounds returned no-go and both were right; the findings below were real defects, not style. Round 1Correctness pass — no-go. The switch was too coarse: turning off every job on an HTTP-only instance also disabled work that must run there, because it refreshes process-local state that requests on that same instance read.
Addressed by the Conventions pass — go with findings. The empty string was treated as "unset" and would have silently re-enabled the scheduler — the likeliest deployment accident of all ( Round 2Correctness pass — no-go. Round 1 had only reviewed the 21 jobs without a
The scan also flagged five jobs that merely set a "warning already logged" flag while doing global work (pay-in registration, fiat sync, monitoring observers). Marking those would have processed incoming funds twice — they deliberately stay global. Verified rather than assumed
Known limitation
Still openOne reviewer finding is not addressed: This PR stays in draft. It is not ready for review while that item and the |
Status: the switch is groundwork, not yet safe to useA full inventory of the codebase (static call graph across all 2066
What this PR contributes stands on its own and is worth merging: the What must not happen is setting |
779a5de to
81a281f
Compare
❌ TypeScript: 2 errors |
6b1e74f to
bfd4264
Compare
|
Seven review passes until both reviewers reported no findings, each pass a pair of independent reviews over the full diff — one on conformance and completeness, one on logic and correctness. The findings that carried weight, and what changed because of them: The monitoring write could lose a value, in three layers. Merging the changed metrics into the stored row narrowed the race instead of closing it: two writers that both read before either wrote still overwrote each other, and the lost value did not come back, because the next run compares against this process's own previous state and finds the metric unchanged. Read, merge and write now happen in one transaction that locks the row. That surfaced the next layer — a writer that waited on the lock holds a value from before that wait, and wrote it back over a newer one — so a changed metric is only taken over if its timestamp is not older. And an absent row cannot be locked at all, so the merge is retried once, restricted to the errors a retry can resolve. Two caches were scoped for a process that never fills them. The exchange trade map and the refund list are written and read by request paths of their own controllers; under The enum members did not follow the repository. Measured rather than assumed: 335 enums use ALL_CAPS members against 6 using PascalCase, two of which were the ones added here. The job inventory gained a scope column. The earlier reasoning — a column would go stale — applies to interval and flag just as much, and those are maintained. All three now come from the decorator arguments. Also: Several comments were rewritten rather than softened, because they claimed things a reader cannot verify here: production measurements, how the code came to be, a reference to a dashboard living elsewhere, an assertion about the deployment topology, and a claim about what the provider discovery sees that is simply not true. Two findings were rejected with reasons: a comment carrying production data is pre-existing code from |
1769bcb to
ca85f1f
Compare
|
Full audit against The earlier review passes checked the diff, not the guide. Reading all 1260 lines against these 119 files turned up two things: 1. The guide contradicted itself on bare 2. Measured clean across the 106 changed non-spec files (added lines only): no |
27adf63 to
45d1744
Compare
Background jobs and HTTP requests share a single Node event loop, and Node runs JavaScript on one thread. A CPU-heavy job therefore delays every request on the same instance: measured on production, a request to /version - no database access, handler time 1 ms - took p95 5.5 s when measured locally against the container, with event loop utilization averaging 84%. The delay followed the cron jitter window exactly, so it was driven by the scheduler, not by traffic. CRON_JOBS_ENABLED=false now makes DfxCronService return from onModuleInit without registering anything, which allows running the jobs in a second instance while the first only answers HTTP. The switch is deliberately coarser than DISABLED_PROCESSES: that one only skips jobs declaring a `process`, and 21 of 131 jobs declare none - among them trade checks, referral credits and volume resets. Those would keep running on both instances, and since cron locks are per-process, nothing would catch the duplicate execution. An unset variable keeps the previous behaviour so every existing environment is unaffected. Any value other than 'true'/'false' throws instead of being coerced: a typo such as 'fals' would otherwise silently re-enable the scheduler on an instance meant to be HTTP-only, and duplicate execution of financial jobs is far more damaging than a failed boot.
Traces answer how long a request took, but not why. A request queued behind a saturated event loop is indistinguishable from one waiting on a slow query, so diagnosing the recent latency issue required taking a V8 CPU profile against the running production process - a one-off snapshot obtained through an intervention in production. MonitorEventLoopService already measures utilization and loop delay correctly, but only writes them to the log, and it runs as a cron job: an HTTP-only instance registers no jobs and would report nothing precisely where saturation matters most. Collection is therefore driven by the OTel metric reader instead of the scheduler, so it runs on every instance. Export reuses the existing OTLP pipeline, so no /metrics endpoint and no additional scrape target are needed. Instrument names follow the Node.js runtime semantic conventions. With OTEL_EXPORTER_OTLP_ENDPOINT unset, no meter is registered and the app boots unchanged, matching how tracing is already handled.
Only a variable that is absent entirely now means "run jobs". An empty value
takes the dangerous path the surrounding comment claims to rule out: an
`CRON_JOBS_ENABLED=` line in an env file or an unresolved `${VAR}` in a compose
file would have re-registered the scheduler on an instance meant to serve HTTP
only, and jobs without a declared `process` would then run on two instances at
once. That accident is more likely than the misspelling the check was written
for, so it fails the boot too.
Also documents the variable in .env.example with an explicit value rather than
the usual empty placeholder, which would now be rejected, and covers the
env -> Config wiring instead of the parser alone: the cron service reads
Config.cronJobsEnabled, and nothing verified that path end to end. The service
test builds its configuration through ConfigService/GetConfig for the same
reason - spreading GetConfig() dropped the prototype getters and left the
global singleton structurally different from a real Configuration.
Shares the telemetry-enabled check between tracing and runtime metrics so the
two halves cannot drift apart, and aligns the sdk-metrics range with the
baseline the rest of the OTel stack pins.
The switch was too coarse. Turning off every job on the HTTP instance also turned off the jobs that must run precisely there, because they refresh process-local state that requests on that same instance read: - resyncDeniedJwtAddresses / resyncDeniedJwtAccounts rebuild the in-memory JWT denylists every 30 s. Without them the lists stay frozen at boot, so blocking an account would no longer revoke its live tokens on the instance that serves HTTP until the next restart. The lookup fails open on an empty set, so nothing would have surfaced the gap. - resyncDisabledProcesses applies the database-driven kill switch and safety mode, which gate several HTTP endpoints. - updateCache holds the transaction specifications behind quote and limit calculation; checkLists and processCleanupMailSecretCache expire local caches that only grow where the traffic arrives. DfxCron therefore takes a `perInstance` flag for work whose effect is confined to its own process, and the switch now skips only the rest. Running such a job on every instance is harmless by construction; anything that writes to the database or drives business forward stays global and is skipped as before. Also moves the two remaining native @Cron decorators to @DfxCron. They were registered by the Nest scheduler directly and bypassed the switch entirely, which made the claim that an HTTP-only instance registers nothing false. Their process guards move into the decorator, where DfxCronService evaluates them anyway, so the bodies lose their early return.
The first pass only reviewed jobs without a `process` parameter, which missed the same class of job among those that declare one. Found by scanning every @DfxCron body for writes to process-local state instead of reading the list again: - payment-link-fee.updateFees holds the fee cache behind LNURL-Pay and payment link quotes. Entries expire after 5 minutes, so on an HTTP-only instance every such request would have started failing five minutes after boot. - tfa.processCleanupSecretCache is the only thing enforcing expiry of mail 2FA codes and enrollment secrets — verify() never checks the expiry date itself. Without it those stay valid indefinitely on the instance serving the requests. - auth-lnurl.processCleanupAccessToken likewise enforces the 30-second window in which a minted JWT is retrievable via the publicly known k1; status() does not check the timestamp. - statistic.doUpdate, binance-pay.updateCertificates, monitorEventLoop and processCleanupAuthCache follow the same pattern. The scan also flagged five jobs that merely set a "warning already logged" flag while doing global work (pay-in registration, fiat sync, monitoring observers). Those deliberately stay global: marking them would have processed incoming funds twice. Known limitation, deliberately not addressed here: /health reads the observer state from memory, and the observers write to the database, so they cannot simply run everywhere. On an HTTP-only instance the endpoint therefore reports the snapshot loaded at boot. Making it re-read that snapshot touches observer semantics and belongs in its own change.
The Spark client starts a bare setInterval in its constructor that runs optimizeTokenOutputs against the wallet every five minutes. It predates the scheduler and bypasses every switch, so with the process split both containers would drive on-chain maintenance against the same seed. It is global work and belongs to the job instance, so it now honours the same flag as the cron jobs. This was the last remaining path by which periodic work could run outside DfxCronService.
Three files were rewritten with LF by tooling, turning one-line changes into roughly 1100 lines of diff noise and guaranteed merge conflicts for any parallel branch touching them. The repository mixes both endings and has no .gitattributes, so the original ending is restored rather than normalised here. monitorConnectionPool and monitorConnectionPoolStatic measure this process's own pool and only log. They are harmless to run everywhere by construction, and without them pool visibility would be lost on exactly the instance whose pool serves the requests - the same reasoning that put perInstance on monitorEventLoop.
Every new cron job now carries a classification decision, and getting it wrong fails silently on the HTTP instance — the JWT denylists were the first example, three more surfaced in later review rounds. The rule therefore belongs in the cron section of CONTRIBUTING, not only in the TSDoc of the flag. Also corrects a comment in runtime-metrics.ts that no longer held: monitorEventLoop became per-instance in the meantime, so it does run on an HTTP-only instance. The reason for the metric stands - a queryable series instead of a log line, and independent of how the scheduler is configured.
CRON_JOBS_ENABLED could express "this process runs the jobs" and "this process runs none", but not "this job belongs to the process serving HTTP". That third case exists: a job maintaining state that only a request path reads, or driving work bound to the connections a process holds open, is worse off in the worker than it is today. The flag also carried its answer in the wrong place - an optional perInstance defaulting to "global" points the silent failure at the dangerous side, because a job wrongly left unmarked stops refreshing the cache that requests on the HTTP process read, without an error anywhere. CRON_ROLE replaces it with three operating modes and CronScope with three job properties. A role runs its own scope plus `both`; `all` runs everything and is the single-process mode, which keeps local development, the test suite and every environment without a separate worker exactly as they are. The variable has no default and an unknown or empty value aborts the boot. Every default is silent in one direction: defaulting to `worker` makes a misconfigured API process run all background work a second time, defaulting to `api` makes a misconfigured worker do nothing at all. Neither raises an error, and duplicate execution of financial jobs is worse than a failed boot. The existing perInstance markings move to scope: `both` for the fifteen jobs refreshing state that both sides read, with two corrections. StatisticService becomes `api` - the hourly aggregation only feeds GET /statistic, so running it in the worker would compute a value nobody there retrieves. BinancePayService becomes `worker` - getCertificates() refreshes lazily on read, so its cache cannot go stale in a process that never runs the job. Registration also logs one line per process stating which split actually applies. A job reaching the scheduler through a dynamically resolved provider or an abstract base class is counted there and nowhere else, so the effective assignment is read from the log rather than inferred from a list. The Spark token optimization follows the same rule from its own timer: it is on-chain wallet maintenance and must run in exactly one process.
The role only decides which jobs a process registers; it cannot decide which process a job belongs to. That answer has to come from the job, and leaving it optional puts the silent failure on the wrong side: a job that forgets the field keeps running everywhere, and one wrongly scoped `worker` stops refreshing the cache the request path reads, in both cases without an error. The alternative - a default plus a list of exceptions - moves the decision into a hand-maintained list the compiler never sees. Such a list grows, goes stale, and pinning it in a test proves the state of the list rather than the property it stands for. Making the field mandatory moves the check into the compiler, where it stays complete for every job added from here on. The price is paid once, here: 116 decorators gain `scope: CronScope.Worker`, which is what they already did and what any job touching only the database or an external system should do. The seventeen exceptions were classified in the previous commit. The change is mechanical and carries no behaviour: with CRON_ROLE=all every job is registered regardless of its scope.
The runtime kill switch works per Process, and a job declaring none cannot be stopped without a deploy. That is the tool the split relies on when a single job misbehaves in the process it was moved to, so the three jobs where stopping one matters get a flag: - DexService::finalizePurchaseOrders writes liquidity orders, the only one of the three carrying real financial risk. - RefService::checkRefs removes rows. - JwtRevocationSyncService::syncDeniedJwtAccounts writes a setting. It is idempotent, but it should run once rather than not at all. The remaining jobs without a flag are the monthly and annual volume resets. Their statement finds nothing left to do on a second run, so a flag would be good practice but is not a precondition for anything here.
Two native @Cron decorators and one bare setInterval had grown alongside DfxCronService and are invisible to it. A scope cannot reach them, so after the split they would run in both processes: the transaction request sync deletes rows, and the Spark timer performs on-chain wallet maintenance against a single seed. Nothing in the repository would have flagged the next one. Both decorators had already moved to @DfxCron; txRequestStatusSync now also carries useDelay: false. DfxCron staggers job starts by default, and for a minute-based expression that spreads them over up to 30 seconds - without the flag the move would have changed when the job runs, not just how it is registered. The daily job reaches no branch of the delay logic and is unaffected. The guard test is deliberately syntactic: it asks whether @Cron( or setInterval( occurs, not whether the code behind it is safe. Its exception list therefore has a natural ceiling, and the one entry - a timer tied to the lifetime of a client object rather than to a schedule - is a decision, not a special case. A third test asserts each exception still matches something, so a leftover entry cannot quietly read as a rule that still applies.
The observers maintain their results in a BehaviorSubject inside the process running them, and eleven endpoints read that field: GET /monitoring/data and the six health paths. Scoping the observers to the worker would leave those answers frozen at the boot snapshot in the API process - the health report included - while scoping them to the API process would move AML, node and bank queries back into the request path, which is the work the split exists to move out. Neither is necessary, because the state is already persisted in full and reloaded at boot. Only the read path did not use it. It does now, with a 30-second process cache in front and one shared read for concurrent requests. The refresh sits before the branching: /monitoring/data takes subsystem and metric as query parameters, and refreshing only the unfiltered branch would leave exactly the filtered answers stale. What this process holds more recently still wins, so a single-process setup is as current as before and a value arriving through the webhook is visible before the next write. The write path needed its own answer. Every process subscribes to its own updates and writes the whole state as a single row, so whichever writes last would drop the other's work entirely - the API process overwriting the observers with its boot state, or the webhook value being overwritten by the next observer run. Only the metrics changed in this process are now merged into the stored row. Binding the write to one role would have been smaller, but it would leave the webhook path silently ineffective rather than working. The reload does not send mail on failure. At boot that notification runs once; on a path taken by every request it would answer a database outage with a flood of mail, during the outage.
processExpiredPayments and checkTxConfirmations both reach PaymentLinkPaymentService.doSave(), which resolves the AsyncMap that PaymentLinkController's waitForPayment holds open and pushes the device activation into the RxJS subject the gateway delivers to its connected clients. Both are process-local, and both are read only from an HTTP path. The worker has no ingress, so no point-of-sale device ever connects to it. Run there, the confirmation would reach nobody: the waiting request hangs until the connection drops, because waitForPayment passes timeout 0 and AsyncMap.wait creates no timer for that, so the promise is never rejected. `Both` is no option: the jobs write to the database and trigger merchant webhooks, which two processes without a shared lock would do twice. The cost is deliberate - the confirmation work, blockchain client calls included, stays in the request process. Moving it would mean waking a waiting connection across process boundaries, and introducing that into the live payment path is a larger risk than leaving the work where it already runs. This is also why a binary cron switch could not express the split: some work is bound to the process holding the open connections.
waitForPayment waited without an upper bound. A client that hangs up says nothing the server can hear, so its entry stayed in the wait map and in waitStates for the lifetime of the process — that is where the growth came from, not from a side effect of it. The wait is now bounded and the waiter clears its own entry on the way out, whichever way it leaves. The endpoints keep their shape: when the bound elapses they answer with the payment as it stands. The device register is no longer mirrored into the service. The gateway owns the sockets, so it is asked what is connected instead of reporting connects and disconnects into a second map that could drift from them — which is what the reference count did when one close path fired twice. Registration is bound to the socket and to every way it can end, including the error path, and a ping/pong sweep drops sockets that stopped answering, since a peer that disappears without closing fires no event at all. The delivery window is taken per device rather than as one minimum across all of them, where the quietest device set the window for everyone.
CronLeaseService already stated under 'What it does not do' that a lease can expire under a running job and let a second process start it. Twelve other places said the opposite — 'at most one process', 'closes that by construction', 'makes it structural', 'the double run the lease exists to prevent'. On a money path that difference decides what an operator does during an incident: told the double run is impossible, running a second worker looks safe. They now all say the same thing. The window is bounded by the lease TTL instead of running until someone reads an alert; what a job scoped worker rests on is a deployment that runs one worker and a job that tolerates a repeat; the lease is defence in depth over those two rather than a replacement for either.
Losing the lease neither stops nor pauses the run that lost it, so the overlap ends when that run ends - up to the two hours the longest job timeout allows. What is bounded is how long a claim left behind by a dead process blocks the job, and how long a second process waits before it may take the job over. Also drops a stop grace period taken from the deployment configuration, an empty test exception list the documentation described as populated, and a heartbeat selector anchored at the one end of the line that carries caller-supplied text.
processExpiredPayments runs in the worker while the expiry timers stay in the process that served the request, and cancelling and completing arrive from request paths, so several processes can read the same payment as Pending and each send the merchant its webhook and cancel the quotes again. A conditional update on the status decides who performs the transition; only the affected row triggers what follows. It holds for any number of processes and for every path into the transition.
The start-up call bypasses the scheduler, so it applied the role but not Process.LATEST_BALANCE_CACHE. Switching the job off still left one run per deployment.
The connection sweep drops its entry before terminating the socket, so a terminate that throws cannot leave the entry behind and stall every following sweep on it. The comments: the payment-link domain has three Worker jobs, not one; the metric export interval is only read when an OTLP endpoint is configured; and the scope on ExchangeController::checkTrades applies to a job that is not registered anywhere, because the class is a controller.
The conditional update committed on its own, so a caller that stopped between it and the quote cancellations left a payment out of Pending with its quotes still open. processExpiredPayments asks for Pending and would never see that row again. The effects that write to the database now run in the transaction that takes the transition, and the merchant webhook stays outside it.
npm resolved the caret range to a version above the one the six exporter packages depend on, so two copies of @opentelemetry/sdk-metrics sat in the tree: the meter provider was built against one and read through the other. The lockfile now holds a single copy.
An editing step rewrote both files with LF, which showed up as a full rewrite of files that carry three changed lines each. Also drops a local scratch file that never belonged in the tree.
The store said there was one writer per process, but the job filling it is leased: with more than one API process, one takes the tick and the rest never run it, so their store stayed at whatever they started with. It now loads on demand through AsyncCache and ages out after a minute, which is what CONTRIBUTING asks of a cache a request path reads; the job is refresh only, and the start-up fill it replaced is gone.
The comment stated how many declarations carry the timeout and called it the longest value in the repository. No test holds either claim, and the next job added makes both wrong without a word.
The comment claimed the activations were closed before the transition, which holds only when the quote reached a state PaymentQuoteFinalStates lists - a completion threshold of TX_RECEIVED does not.
The lease decides who runs the job, not how long two runs overlap; in the shutdown case that is the container's stop grace period.
AsyncCache hands a running update to everyone who asks for the entry while it is in flight, which is what keeps a restart from starting one aggregation per request. The read relies on it and does not implement it, so it is asserted rather than assumed.
The device delivery kept a per-device high-water mark and moved it to the newest `updated` it had read. A payment is stamped by the statement that writes it and becomes readable only when that write commits, so a mark past the stamp of a row still in flight asks for `updated > since` and never sees that row again. Two rows stamped alike are the plainest case; any write outlived by the read beside it does the same. A span measured against the present cannot skip a row that way, and it bounds the read for a connection of any age — the mark only did so while payments kept arriving. What the span admits twice, a record of the states already sent answers for, per payment rather than one slot per device, and that record is dropped with the payments that leave the window.
The transaction tests counted the effects but never checked what they ran on, so removing the manager from the calls left them green — and the manager is the whole property: an effect on any other one is a statement that commits whether the transition does or not.
The read that tells a connected device its payment is through was bounded by `updated`, a column every write stamps. Two rounds of review found the same failure through it twice: a mark advanced past a row that had not committed, and then a span against the present that a transaction outliving it walks a row straight past. Both end the same way — the device is never told, and nothing can bring the row back into the read. `expiryDate` is given at insert and never moved afterwards, so a late commit can only make a row appear later, never make it skip. The read now selects on it, one cutoff for every connected device, reaching back past the payment's own end AND past the configured delay before an expiry is acted on — read from the configuration rather than assumed, so raising it cannot silently drop the expiry transition out of the read. The connection time goes with it. What a device is owed follows from the payments, not from when it happened to connect, so a device that reconnects is owed what it was owed before: the delivery record now ages out on the same cutoff the query uses instead of being dropped with the connection, and a device whose entries have all gone leaves with them.
Four summaries described the lease as held "for the duration of its run" or "for as long as the job runs", and one described the renewal as keeping the claim alive. Each is the short form of a passage that goes on to take it back: the claim can lapse while the job is still working, and losing it stops nothing. A reader who takes only the short form takes away a stronger property than the design has. Shortened rather than qualified, because this is the fourth round in which the same class of sentence has come back. What the lease does is claimed before the start; what happens after that is in "What it does not do", where it already was.
Round seven found two places where a boundary drawn in an earlier round stops one statement short of where the damage is. In `handleQuoteChange`, the activations of a final quote were closed BEFORE the status moved. If the transition then failed, the payment stayed `Pending` with its activations closed for good — and nothing brings that back: the quote is already final, so `checkTxConfirmations` does not return to it, while `processExpiredPayments` only ever asks for `Pending` and would expire a payment that was in fact paid. The close now travels with the transition on the path that has one, and stays where it was on the paths that do not. In `stateRow`, two writers converging an old snapshot row onto `id: 1` could lose the first one's work. Both miss `id: 1`, both queue for the old row, and the second wakes up after the first has created `id: 1` and committed — merging from the old row then writes a state that predates it. Since this path only writes what changed, the lost metric does not come back on its own. The lookup now asks a second time after the wait. Two statements were also promising more than they hold. The delivery grace said a late commit can never make a row skip; that is true of WRITES, and the read still ends somewhere — a transition after that end is missed like any other. The span now reaches an hour instead of ten minutes, chosen against what actually delays the transition: the worker being gone, which the silence alert only reports after seventeen. And the shutdown release is best effort, not a guarantee: a claim still being taken is not in `inFlight`, so an exit in between leaves the row to lapse on its TTL — the bound that always applies. Each of the four is pinned by a test whose counter-proof was seen red.
Round eight, seven findings, none of them in the lease or the role assignment. The delivery job carried a `process` flag like any other job. It is not a job in that sense: it is the bridge that carries a result from the process that wrote it to the process holding the connection. Off in the single-process setup nothing happens, because `doSave` delivers directly there — so the flag looks harmless right up to the moment it is not. After the split, switching it off silently cuts delivery to everything attached to the other container, and no alert sees it: every process still reports its role and a usable lease. The flag is gone, and with it the enum value this branch had added for it. The same reasoning already governs the role heartbeat and `checkConnections`. The delivery record is written BEFORE the send, and that ordering is now stated rather than implied — together with what it costs and why the alternative costs more. The gateway no longer leaves a socket in the map that threw on `send`: a device unreachable through it would otherwise keep being selected, and counted as delivered. Two comments promised a reconnecting device everything it was owed; they now say "within the span the read covers". Three more: the heartbeat's error form was abbreviated in the comment that declares it an interface, while the alert reads the full line. Two comments still said the alert reports a double run — it reports a wrong ROLE, which is what the logs can distinguish. And the spark job caught its own error to log it again, which is the redundant try-catch CONTRIBUTING names by example. The transaction proof had a hole the review was right to name: the existing tests show the effect services are HANDED the manager, which a service that ignores it would also pass. The new spec asserts the other direction — given a manager the write goes through that manager's repository and never through the injected one, and without one it goes through the injected one.
Round nine found the delivery could lose a command silently, and the code said the opposite in two places that could not both be true. The record of what a device has been told is keyed by DEVICE and outlives the connection — `pruneDeliveries` keeps it that way on purpose, so a reconnecting device is not told twice. The write path meanwhile claimed that a failed send would go out again "on the next tick under a new connection". It would not: the record already said delivered, and the periodic delivery that would repair it takes the early return. A `send` that threw was therefore lost for good, on the one path built to prevent exactly that. The gateway now hands the command over through a sink that answers whether it reached a socket, and the record is written only on `true`. That is what the RxJS subject could not do: a subject carries a value one way and swallows what the subscriber does with it, including a throw. It had one consumer, so it is gone rather than wrapped. The cost of this order is a repeat if the process dies between the send and the record. Repeating is what `waitState` makes harmless; losing is not. Also: `docs/cron-jobs.md` counted 118 of 140 jobs as flagged with six deliberate exceptions. Removing the delivery job's flag last round made that 117, 23 and seven — recounted from the table rather than adjusted by one, and the delivery job now appears in the list of deliberate exceptions with its reason instead of silently among the omissions.
Round nine moved the delivery record behind the send so a failed one would be retried. Round ten found the half of that which `send` cannot report. `ws.send()` throws only while the socket is still CONNECTING. On one that is CLOSING or CLOSED it takes the call without a word and raises the failure asynchronously through `'error'` — by which time the sink has already answered `true` and the delivery has recorded the command against a state it will not send again, not even when the device reconnects. That is the same silent loss as before, reached through the one door the try/catch does not cover. So the state is read first and only an open socket is written to; one that is not open is dropped, exactly like one that throws. What remains is a socket closing BETWEEN the check and the send — one synchronous call wide, and the delivery survives that the way it survives a restart, because the payment stays inside the read until its own end. What it does not survive is being told `true` for a socket that was already gone. Three smaller ones. Two comments still described the RxJS subject that round nine replaced — a later reader would have looked for a subscription instead of the sink and missed that it answers. `PaymentLinkFeeService` justified its `Api` scope with "one of the Api-scoped payment crons"; it is the only one in the domain, and it writes the cache rather than reading it. And two imports were not alphabetical.
The cross-repo review found a blocker in the rollout, not in a single PR — the kind only that review can see. Between the production release of this application version (step 5) and the release that brings the lease alert (step 9), production runs with the lease in force and nothing watching it. `guardAcrossProcesses` does not look at the role, so under `CRON_ROLE=all` all 123 non-`BOTH` jobs go through the claim. An unreachable table — a missing grant, a migration that did not run, a database hiccup — made every one of them skip, payouts included, while every deployed rule stayed green. Under `all` the deployment runs ONE process. That is the shape the API had before this branch existed, with no lease at all. Skipping there made the lease strictly worse than its own absence, which cannot be the right answer for a mechanism that is supposed to add safety. So under `all` the job now runs, and the failure is logged loudly and carried out in the heartbeat. Under `api` or `worker` the skip stands: there the lease is the only separation, and running anyway is the double run it exists to prevent. Four more from the same round. The heartbeat is now written once at boot. It fires on fixed marks without jitter, so a process that comes up at :11 and misses :10 wrote nothing until :20 — against a twelve-minute window, an ordinary deploy could produce the critical "worker is silent" alarm. The gateway drops what it believes it told a device when a socket reports `'error'`. That is how `ws` reports a send on a socket that closed between the state check and the call, and the comment that dismissed it was wrong: it claimed the record would age out and the delivery retry. It cannot — the record ages on the same cutoff the query uses, so "record gone" and "payment still in the read" exclude each other. `PaymentLinkFeeService.getMinFee` loads on demand instead of answering `undefined`. CONTRIBUTING requires that of every cache read in a request path, and this PR made it bite: the refresh is now `Api`-scoped and leased, so among several API processes only one wins it per tick. And `SET LOCAL lock_timeout` is scoped to the whole migration batch, not to the statement below it — the comment said the first and argued the second.
Round twelve found that the previous round's fix left two holes and eleven sentences describing the behaviour it replaced. The fail-open path called the task directly. That skipped the two things the healthy path does between the claim and the call, and neither is about the lease: the run never entered `inFlight`, so `shutdown` gave it no grace period and reported nothing was running; and it started even if shutdown had begun during the failed claim — a WIDER window than the healthy one, because a failing attempt runs to the database timeout. A payout could be cut off part-way with the shutdown log saying the process was idle. Both paths now go through `track`, which owns exactly those two: the late shutdown check and the `inFlight` entry. What the claim-holding path adds on top — stop renewing, hand the claim back — it passes in. Eleven statements still described the lease as fail-closed everywhere. Five in this repo: the boot error line a reader sees in production, the `run` doc twelve lines above the code doing the opposite, `takeFailures`, `reportRole` and `onModuleInit`. They now say what depends on the role and what does not — under `api`/`worker` the jobs stop, under `all` they run without a claim, and both are states to fix, which is why the heartbeat carries either out. The boot line branches too: under `all` it would have told an operator the opposite of what the process does.
Three things the on-demand load in getMinFee left open, all of them consequences of the same move — a value that used to be produced by one process for itself is now produced in one process and read in another. onModuleInit ran the full sweep in EVERY process. It is a Nest lifecycle hook, not a cron job, so the `Api` scope on updateFees never reached it: at boot the worker queried gas prices for eight EVM chains plus Bitcoin and Firo estimates to fill a map nothing in that process reads — exactly what the scope exists to prevent. The hook now asks whether this process serves requests, which is the property that makes a warm cache worth anything; the role-to-scope table stays in one place, in runsInThisRole. The load carried no LOC guard, while the job it complements does. Locally there are no node connections to ask, so the cache stays empty by design and every request would have run into the timeout of every one of those calls. It answers undefined again, as it did before this method loaded anything. And it started one fetch per caller: the cache is written when a load RESOLVES, so a burst of quotes for the same chain all saw the same empty entry. That is not hypothetical — updateFees is leased, so among several API processes only one wins it per tick and the others rely on this path for a whole minute. Concurrent callers now share the load in flight; a failed one is dropped in `finally`, so the next caller retries instead of inheriting it. Proven by 12 tests, each of the three guards checked against its own removal.
c026a1b to
09fe4f6
Compare
The comment explained why waiting for the record to age out does not repair a socket that closed between the state check and the send — and framed it as a correction of a previous version of itself. The reason is the same either way, and the reader of a merged file has no previous version to compare against.
Important
CRON_ROLEmust be set in every environment before this is deployed. The boot abortswithout it — deliberately, see below — so an environment missing the variable fails on the next
deploy, whatever that deploy was for. The two repositories have separate pipelines with no
dependency between them: merging here triggers a deploy without checking whether the variable is
already in place. The configuration change is a separate, already open PR in the infrastructure
repository; it is inert for the currently running image.
Warning
This PR bundles four subsystems, and a revert is all-or-nothing across all four. That is a
deliberate decision, recorded here so it does not surprise anyone at merge time. See
What this PR bundles directly below before approving.
What this PR bundles
The title says process separation, and that is the reason the branch exists. It is not all the
branch contains. Four independently revertable subsystems ended up on one branch, and because they
are one branch they are also one revert: pulling any of them out after the fact pulls out the other
three.
CRON_ROLE,CronScope, mandatoryscopeon 140@DfxCrondeclarations, boot log, role heartbeatcron_leasetable, migration, entity,CronLeaseService, shutdown handler inmain.tssrc/runtime-metrics.ts, new dependencies, newOTEL_METRIC_EXPORT_INTERVALMonitoringServicereworkpessimistic_writemerge on writeDashboardFinancialServicereworkapijob,LatestBalanceStoreTwo of the five (metrics pipeline, and the lease insofar as it changes the schema) would stand on
their own. They are not being split out: the operator has decided against a follow-up PR, so the
bundling is stated instead of removed. What that means concretely:
them, whether or not the reason for reverting had anything to do with them.
wrong in a two-process deployment — so a revert here has to be accompanied by a rollback of the
role configuration, not just of this code.
cron_leasetable. The migration is not reverted by reverting thecode; the table is simply left unused.
The safe rollback is not a revert of this PR but
CRON_ROLE=all, under which every processregisters every job. That is the mode this branch is designed to be merged in.
Why
Background jobs and HTTP requests share a single Node event loop, and Node runs JavaScript on one
thread. A CPU-heavy job therefore delays every request in the same process.
Measured in production while investigating slow API responses:
/version(no DB access, 1 ms handler), measured locally against the containerA p50 of a few milliseconds next to a p99 in the seconds is the signature of a blocked loop, not of
slow work: the same distribution reproduces on a route touching neither the database nor any
external service, and it persists when the network path is bypassed entirely. Database, host and
network were each ruled out separately — during one 6.1 s freeze the database had zero active
queries, and the connection pool never had a single waiting request.
The cause is the scheduler, not traffic. Grouping the delay measurements by second-within-minute
over six hours (2,154 windows) gives stall rates of 5.8 / 52.8 / 73.3 / 81.8 / 50.0 / 21.2 % — a
factor of 14, following the scatter curve of the job start delay exactly. Request traffic in the
same grid varies by a factor of 2 and peaks elsewhere.
What changes
This PR is the application half of running the same image twice: one process serving HTTP, one
running the background work. Under
CRON_ROLE=allno existing@DfxCronis left out on account ofits scope — new are the role heartbeat, the Spark wallet maintenance job,
refreshLatestBalance,the payment delivery job, the WebSocket liveness sweep and the registration logs.
1.
CRON_ROLEdecides which jobs a process registers, andCronScopesays which process a jobbelongs to. Three values each, because two would not be enough in either direction:
allis needed because otherwise no value covers every job — local development, the testsuite and any deployment without a separate worker would each lose something.
apiis needed because some work is bound to the process holding the open connections —state a request path reads in this process, which a job running elsewhere cannot maintain.
Delivering to those connections is a different matter and is no longer done from an
api-scopedjob; see The api scope and the lease for
why that separation had to be made.
The variable has no default and an unknown or empty value aborts the boot. Every possible default
is silent in one direction:
workerwould make a misconfigured API process run all background worktwice,
apiwould make a misconfigured worker do nothing at all.2.
scopeis mandatory on every@DfxCron. A wrong classification fails silently — a jobwrongly scoped
workerleaves the cache it maintains empty in the process that reads it, with noerror anywhere. A default plus an exception list moves that decision into a hand-maintained list
the compiler never sees; such a list grows and goes stale, and pinning it in a test proves the
state of the list rather than the property it stands for.
Counted from the source tree rather than carried forward: 140
@DfxCrondeclarations — 119worker, 5api, 16both, across 98 files and 34 areas; 118 carry aprocessflag, 22 donot, five of those deliberately. 139 of the 140 have a registration path.
docs/cron-jobs.mdcarries the assignment per job, generated from the decorators, and both of its distribution tables
sum to 140.
3. Cases where a scope alone would not have been enough:
GET /health*and/monitoring/data). Scoping theobservers to the worker would freeze eleven endpoints at the boot snapshot in the API process,
the health report included; scoping them to the API process would move AML, node and bank queries
back into the request path. The state is already persisted in full, so the read path takes it from
there with a 30-second process cache. The write path needed its own answer: every process writes
the whole state as a single row, so only the metrics changed in this process are merged into the
stored row — otherwise whichever writes last drops the other's work.
the database. That property was measured (23 ms median, 1,989 ms p95 before it existed) and is
kept: building the response becomes its own minute job scoped
api, and the aggregation stays inthe worker.
@Crondecorators and a baresetIntervaldriving on-chain wallet maintenance. All three would have run in both processes.A test forbids the patterns, and carries no exceptions.
4. Four jobs gain a
processflag so a single misbehaving job can be stopped at runtimeinstead of by deploy, and the worker reports itself under its own service name so its outgoing
calls do not read as API traffic.
Cross-process lease
An architecture review named the load-bearing assumption of the original design: "there is exactly
one worker, and the configuration is right". It was held up by convention, a runbook sentence and
an alert that reports a double run 15–25 minutes later. For a path that moves money, detection
is the second-best answer — and
LockClasskeeps its state in a field in process memory, so itcannot see a second process at all.
Jobs scoped
workerorapihold a lease in thecron_leasetable for the duration of theirrun. A single statement decides it: the upsert takes the row over only if it has expired. Two
processes racing are serialised by the primary key and one of them gets a row back.
What that is worth, stated precisely. The lease does not exclude a double run, and it does
not bound how long one lasts. A job runs once because the deployment runs one worker and because
the job tolerates being run again. What the lease adds underneath those two properties is that a
second process has to take the claim before it may start the job: while the holder keeps
renewing, a wrongly configured second process — a missed recreate, a second worker from
--scale,two processes left on
allafter a rollback — does not start it at all. If the holder stopsrenewing while it is still working, the claim lapses, the second process starts, and the first runs
on to its own end; nothing here shortens that, because a running function cannot be aborted from
the outside in JavaScript and a cooperative check at every write is the same work as the fencing
token this does not carry.
CronLeaseServicesays so in its own "What it does not do".What the lease does bound is the waiting: a claim left behind by a process that was killed
blocks the job for the lease TTL rather than until somebody reads an alert, and that same span is
the longest a second process waits before it may take the job over.
Scope
bothis deliberately exempt. Those jobs maintain state a request path reads in theirown process; a lease over them would starve whichever lost the race and freeze that state. Their
safety comes from a different property: running twice has to be harmless by construction, which is
what CONTRIBUTING requires of them.
Why a lease and not an advisory lock.
pg_advisory_lockis bound to the connection and wouldhold a pooled connection for the whole runtime of the job. 67 jobs declare a timeout measured in
minutes; that is a real risk to a connection pool sized by
SQL_POOL_MAX. An expiring rowcosts one short query each to take, extend and release.
The lease is 60 seconds, renewed every 20, and unrelated to the job's timeout. The expiry bounds
how long a claim outlives an owner that can no longer speak for itself — SIGKILL, an OOM kill, a
lost machine. That is a property of the failure mode, not of the work, and deriving it from the
job's own timeout got it backwards:
timeoutis measured in seconds and nineteen@DfxCrondeclarations carry 7200, so a process killed mid-run used to block its own successor from that job
for up to two hours, silently.
Shutdown is the other half. Nothing in this repository ever asked for a shutdown hook, so
SIGTERM ended the process instantly and the release in the
finallynever ran.CronLeaseService.shutdownnow waits up to ten seconds for the runs this process holds, so theirnormal release hands the job to the successor. A run still going after that keeps its lease:
taking it away would let the successor start the same job while this process works on it for the
rest of the grace, which is the outcome the lease exists to make rare.
It hangs off a SIGTERM/SIGINT handler rather than
app.enableShutdownHooks(), and that isdeliberate. The idiomatic call is global: it would also start running the nine
onModuleDestroyimplementations this application carries but has never executed, and Nest runs those before the
lease hook — they empty the strategy registries that PayIn, PayOut and DEX jobs resolve from. On
its own that is harmless, since the process was about to die. Next to this change it is not: the
wait deliberately keeps in-flight jobs alive longer, so a running payout would gain time to fail on
an emptied registry rather than simply be cut off. A test pins that the idiomatic call stays out.
An unusable lease table is reported rather than silent. Without it every worker- and api-scoped
job is skipped on every tick — the right behaviour, and what CONTRIBUTING asks for, but the skip
used to look exactly like a job with nothing to do. The role heartbeat is scope
bothandtherefore exempt from the lease, so it kept reporting a healthy process while everything it counts
sat out; and it counts registered jobs, which cannot see this at all. The lease layer now reads
the table at start-up and carries a health flag that stays false until an operation gets through;
the heartbeat writes the same line at error level with the reason appended when it is bad. The same
line, because the role alert matches on its shape.
What it still cannot do — and the code says so. If the holder stops renewing while a job runs,
the claim expires and a second process can start the job while the first is still working, for as
long as that first run takes. Full fencing would need a token on every single write. What the lease
shortens is the wait, not the overlap.
With the database unreachable the job does not run. A job that moves money must not proceed on
the assumption that it is probably alone — and that is exactly when the assumption is least safe.
The api scope and the lease pull against each other
PaymentCronServicewrites to the database and triggers merchant webhooks, which argues for oneprocess, and it used to be the only thing releasing the callers waiting on the process that ran it,
which argues for every process. A single scope cannot satisfy both, and the lease made that
visible: whichever process lost the claim left its callers waiting for nothing.
The resolution taken is to split the job rather than the scope. The writing runs under the lease
(
Worker), anddeliverPaymentUpdatesdelivers from the state those writes leave behind, scopedBoth, in every process, without a lease — it writes nothing and calls nothing outside its ownprocess, which is what allows it to run everywhere. An
api-scoped job losing the race is stilllogged at error level, unlike a worker job, which loses it every cycle by design.
Leaving
Pendingis decided by the update statement rather than by a status read, because theexpiry timers stay in the process that served the request while the expiry job runs in the worker,
and cancelling and completing arrive from request paths as well. Only the caller the database lets
past
Pendingsends the merchant its webhook.Schema
migration/1785600000000-AddCronLease.jscreatescron_lease. Two things worth stating:PK_a12c181c2b26f33be13d55a15af—PK_plus the first 27 characters ofsha1('cron_lease_name'), which is what TypeORM's own naming strategy produces. A hand-pickedname would not be recognised by a schema comparison, which would then offer to create the
constraint. A new test recomputes every primary key declared in a
CREATE TABLEacross allmigrations (102 of them, all matching) and rejects any spelled-out constraint name.
acquiredandexpiresaretimestamptz. They are compared againstnow()in raw SQL, and avalue without a zone on one side of that comparison resolves through whatever time zone the
session carries: the same row expires an hour late or an hour early across a daylight saving
change. An hour late is a job that runs nowhere, an hour early is two processes running it.
src/shared/models/cron-lease/cron-lease.entity.tsmirrors the table. The service never readsthrough a repository — the claim is a single
INSERT .. ON CONFLICT .. WHEREthe query buildercannot express — but a table that exists only as DDL is invisible to the entity model, and the next
generated migration would read that absence as an instruction to
DROP TABLE "cron_lease". A testbuilds the entity metadata without a connection and checks it against the migration file.
Deployment
The order matters and is not optional:
CRON_ROLE=allmust be set in every environment before this PR is deployed. The boot abortswithout it, so an environment missing the variable fails on the next routine deploy — for the
currently running image the variable is unknown and therefore inert.
cron_leaseruns with it.CronRole all: registered 139 of 139 jobs (worker: 119, api: 4, both: 16)— and health and dashboard endpoints answer unchanged.registered 135 of 139in the worker andregistered 20 of 139in the HTTP process.Rolling back never means reverting this PR: under
CRON_ROLE=allits content is today's behaviour,so what gets reset is configuration. See What this PR bundles for why a
revert is the expensive option.
Testing
npm test, plus targeted runs on the affected suites;tsc --noEmit, ESLint and Prettier clean.Coverage added by this branch:
a throw rather than a silent default
env -> Configwiring the cron service actually reads, not just the parserallregisters everything, each role drops the other's scope,bothsurvives in all three
timeout: 7200still claims for 60seconds — the regression that made a deployment block a job for two hours
return before the job does,
main.tsis pinned to wire it to the signal at all, and pinned NOTto reach for
app.enableShutdownHooks()failure occurred, and reported healthy again once a claim gets through
cron_leaseentity against its own DDLqueries, the merge prefers whichever value is newer, the write path keeps metrics another process
wrote, the row is read under a write lock in the same transaction it writes in, an older
measurement is not put back over a newer one, and the merge is retried only on errors a retry can
resolve
id: 1is answered fromthe row that exists, and the write seeds
id: 1from it rather than with a partial stateapiandall, off when the process flagis off, and a failure reported instead of left as an unhandled rejection
worker@Cron(,@Interval(,@Timeout(andsetInterval(, including a check thatits exception list still matches something
Every fix in the review rounds below was verified by putting the defect back and watching the test
fail, then restoring it.
Known discrepancies, recorded rather than fixed
ExchangeController::checkTradesis never registered: its class is listed undercontrollers:in
ExchangeModuleand nowhere underproviders:, andDiscoveryService.getProviders()does notreturn controllers, so the scan never sees the decorator. This predates the process split — the job
has never run.
TransactionController::checkListslooks like the same case but is not:HistoryModulelists that class under bothcontrollers:andproviders:, so the job isregistered, on the provider instance, which is a different object from the controller instance the
request handlers use.
CitreaBaseStrategy::checkPayInEntriesis declared on an abstract class, so it is registeredonce per concrete subclass rather than once per declaration. There is currently one subclass.
16 jobs carry no
processflag. That is pre-existing, named indocs/cron-jobs.mdas an omissionrather than hidden, and retrofitting it means introducing 16 new kill switches — a decision about
those jobs, not about this PR.
What should happen to any of these is a decision about the jobs, not about this inventory.
Review rounds
Round 1 — rebased onto the current
develop, conflict-free, CI 12/12 green.Round 2 — role heartbeat added.
DfxCronService::reportRolewritesCronRole <role>: heartbeat, N jobs registeredin every process every ten minutes. The reason liesoutside this repository but the line originates here: the Grafana rule meant to report a wrong role
assignment used to read the boot line, which is written exactly once. On a healthy system a
counting window over it reports permanently from the day after the last deploy, because the line
falls out of the window while the container keeps running. And the most expensive state is
precisely the one without a restart: if the recreate for a configuration change does not happen,
the HTTP process keeps its old role while the worker takes over the same jobs. Without a restart
there is no new boot line, so the rule could not see it structurally. Scope
bothso the lineappears in every process, with the role in the line; no
processflag, because a watchdog thatcan be switched off looks, switched off, exactly like the failure it reports.
useDelay: false,because the alert reads a 12-minute window and the jitter is adjustable from outside through
CRON_JOB_DELAY— a watchdog must not have its timing tuned by a knob meant for spreading load.Round 3 — seven review points, three implemented, four re-measured and declined with the
measurement stated. Implemented: the heartbeat tests were reading decorator metadata and would have
stayed green if the scan never saw the method, so the test now hands the scan its own service
instance the way Nest does and expects the heartbeat in the count; the guard test's
setTimeoutgap was documented with its evidence; and the claim "behaviourally identical to today under
CRON_ROLE=all" was narrowed to what is actually true. Declined: thegit diff --checkwhitespacereport (all files are CRLF and
.prettierrcsetsendOfLine: auto; two consecutivedevelopcommits report the same), the
PaymentCronServicescope (see above — the concern was right and isnow addressed by the lease and by reporting the lost race), the 16 missing flags, and the
"environment updates missing" point (they are in the infrastructure repository, because that is
where
CRON_ROLEis set).Round 4 — the cross-process lease, described above.
Round 5 — a five-instance review of this PR and its three infrastructure counterparts. Nine
findings here, all fixed on this branch: the lease TTL derived from the job timeout (up to a
two-hour outage per deployment) and the missing shutdown path; an unusable lease table looking like
a healthy process; a primary key name that violates the deterministic-naming rule; the missing
entity; timestamps without a time zone; the Spark maintenance timer and the statistic start-up fill
both running outside the scheduler and therefore outside the lease; the
apiscope contradictingthe lease; the monitoring read path depending on a row with
id: 1; and this bundling, which isnamed here rather than split out.