diff --git a/.env.example b/.env.example index 68300ecaac..1a7bcb5de0 100644 --- a/.env.example +++ b/.env.example @@ -292,6 +292,11 @@ S3_ADMIN_SECRET_KEY= # OpenTelemetry trace export (OTLP/HTTP). Point this at an OTLP collector to # enable distributed tracing; leave empty to disable. e.g. http://localhost:4318 OTEL_EXPORTER_OTLP_ENDPOINT= +# Metric export interval in milliseconds. Optional: unset leaves the SDK default (60s). +# A shorter interval costs a full collect-and-export of every instrument on the event loop this +# split is meant to keep free. It is only read when OTEL_EXPORTER_OTLP_ENDPOINT is set; with an +# endpoint, an invalid or non-positive value aborts the boot, and without one it is never looked at. +# OTEL_METRIC_EXPORT_INTERVAL=60000 FIXER_BASE_URL= FIXER_API_KEY= @@ -343,3 +348,9 @@ REALUNIT_W2W_GAS_LOW_BALANCE_THRESHOLD=0.05 REQUEST_KNOWN_IPS= CRON_JOB_DELAY= + +# Which jobs this process registers: 'all', 'api' or 'worker'. Mandatory — a missing, empty or +# unknown value fails the boot on purpose, because every possible default lets a misconfiguration +# run silently: one process would do the background work twice, or not at all. +# 'all' is the single-process mode and the right value unless a separate worker process exists. +CRON_ROLE=all diff --git a/.env.local.example b/.env.local.example index c81eb259e0..9e7143ff11 100644 --- a/.env.local.example +++ b/.env.local.example @@ -61,3 +61,8 @@ MAIL_PASS=dummy-password-for-local-dev # compared while REALUNIT_W2W_GAS_WALLET_PRIVATE_KEY/_ADDRESS are unset, so the # value only has to satisfy the boot check. REALUNIT_W2W_GAS_LOW_BALANCE_THRESHOLD=0.05 + +# Which half of the application this process runs: 'all' registers every cron job, which is +# what a local single-process setup wants. There is no default — config.ts rejects a missing or +# unknown value and aborts the boot, so this line is required for `npm run setup` to work. +CRON_ROLE=all diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 262cd00f6d..405c79d494 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -517,12 +517,14 @@ Use `@DfxCron` (custom wrapper with built-in locking, process control, and error ```typescript // GOOD: @DfxCron handles everything -@DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAYMENT, timeout: 1800 }) +@DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAYMENT, timeout: 1800 }) async processPayments(): Promise { // no @Lock, no DisabledProcess check needed } -// ONLY with bare @Cron: manual @Lock + DisabledProcess required +// What the wrapper takes off your hands — NOT an alternative you may pick. +// A bare @Cron carries no scope, so it registers in every process; a guard test +// rejects it (see "Register periodic work through @DfxCron" below). @Cron(CronExpression.EVERY_MINUTE) @Lock(1800) async processPayments(): Promise { @@ -531,14 +533,87 @@ async processPayments(): Promise { } ``` -Declare a `process` flag unless the job maintains the disabled set itself. Without one the job -runs unconditionally and cannot be switched off without a deploy. - -[docs/cron-jobs.md](docs/cron-jobs.md) lists every scheduled job with its interval and flag. +Declare a `process` flag unless the job maintains the disabled set itself, or unless something +outside the process infers health from the job still running — a watchdog that can be switched off +looks, once it is off, exactly like the failure it watches for. Without a flag the job runs +unconditionally and cannot be switched off without a deploy. + +A job scoped `worker` or `api` additionally takes a **lease in the database** before it starts +(`CronLeaseService`). The in-process lock cannot see a second process at all — a missed +recreate, a second worker from `--scale`, two processes on `all` after a rollback — and the lease +is what such a second process has to get past before it may start the job. + +It does **not** make a double run impossible, and it does **not** bound how long one lasts; +nothing in this repository should claim either. If the holder stops renewing while it is still +working — an unreachable database, a blocked event loop — the claim lapses and a second process +can start the job, while the first runs on to its own end, which for the longest-running jobs is +hours. What the lease does bound is the waiting: a claim left behind by a process that was killed +blocks the job for the lease expiry rather than until someone intervenes. What a `worker` job +actually rests on is the deployment running one worker and the job tolerating a repeat; the lease +is defence in depth over those two, not a substitute for either. `CronLeaseService` states the +limit under "What it does not do" — keep any wording here consistent with it. Jobs scoped `both` +are exempt by design: they must run everywhere, which is why running them twice has to be harmless +by construction. + +[docs/cron-jobs.md](docs/cron-jobs.md) lists every scheduled job with its interval, flag and scope. **Adding, removing or re-scheduling a job must be reflected there in the same PR.** Prefer longer intervals (15min) over aggressive polling (1min). Only use short intervals when truly needed. +#### Which process a job belongs to + +The API can run as more than one process from the same image — one serving HTTP, one running the +background work — and `CRON_ROLE` decides which of them a process is (`api`, `worker`, or `all` +for a single-process setup). **Every cron job must declare which process it belongs to**, and the +compiler enforces it: `scope` is a mandatory parameter of `@DfxCron`. + +```typescript +// Worker: writes to the database, moves money, or calls an external system in a way that +// changes state. The normal case. +@DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAYMENT }) +async processPayments(): Promise {} + +// Both: the effect is confined to the process it runs in — refreshing an in-memory copy of +// global state, expiring a local cache, measuring this process. It runs everywhere, because +// requests on the API process read what it maintains. +@DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.BOTH }) +async resyncDeniedJwtAccounts(): Promise {} + +// Api: maintains state read only from a request path. Not for delivering to the connections +// this process holds open — that job is leased too, so it would run in one process while the +// connections are spread over all of them. Deliver from stored state under `Both` instead. +@DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.API, process: Process.UPDATE_STATISTIC }) +async doUpdate(): Promise {} +``` + +Ask: *does a request handler read state this job writes?* If both a request path and a job read +it, the answer is `Both`; if only a request path does, `Api`; otherwise `Worker`. Getting it +wrong fails silently — the state simply freezes at boot wherever the job does not run. The JWT +denylists are the cautionary example: frozen, they fail open and a blocked account keeps its live +tokens. + +A job scoped `Both` runs in every process without a shared lock, so running it twice must be +harmless by construction. If it writes to the database, sends mail, or calls a paid external API, +it is `Worker` — and if a request path needs its result, that result belongs in the database, not +in process memory. + +#### A cache read by a request path loads itself + +**A cache read in a request path must load on demand** — through `AsyncCache`, `CachedRepository` +or a lazy load of its own. A cron job may refresh it, but must not be the only thing filling it. + +This is the rule that makes a wrong scope harmless: a cache that loads itself is correct in every +process, whichever scope its refresh job carries. `AsyncCache` and `CachedRepository` already work +this way; the jobs scoped `Both` are precisely those that do not. + +#### Register periodic work through @DfxCron + +`scope` only reaches jobs going through `@DfxCron`. A native `@Cron` or a bare `setInterval` is +invisible to it and therefore runs in every process — for anything writing to the database, that +means twice, without a shared lock. A test enforces this, and it carries no exceptions: everything +it matches on is gone from the repository. What it does not match on — a repeating `setTimeout` — +its own comment names one by one, with the reason each is left alone. + ### Await Discipline ```typescript @@ -818,13 +893,13 @@ const isValid = await this.validateIban(iban).catch(() => false); ```typescript // BAD: @DfxCron already handles errors -@DfxCron(CronExpression.EVERY_HOUR) +@DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER }) async process(): Promise { try { ... } catch (e) { this.logger.error(e); } // redundant } // GOOD -@DfxCron(CronExpression.EVERY_HOUR) +@DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER }) async process(): Promise { // just do the work } @@ -983,8 +1058,8 @@ Endpoints that block by design: | Path | Blocks until | `wait` segment | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | -| `GET /v1/lnurlp/wait/:id` | the payment resolves: completed, canceled, expired — or, for `MULTIPLE`-mode links, when a quote reaches the configured completion threshold (the payment itself may stay `Pending`) | yes | -| `GET /v1/paymentLink/payment/wait` | the same, for the authenticated payment-link flow | yes | +| `GET /v1/lnurlp/wait/:id` | the payment resolves: completed, canceled, expired — or, for `MULTIPLE`-mode links, when a quote reaches the configured completion threshold (the payment itself may stay `Pending`); bounded at 60 s, after which it answers with the payment as it stands | yes | +| `GET /v1/paymentLink/payment/wait` | the same, and under the same bound, for the authenticated payment-link flow | yes | | `GET /v1/lnurlp/:id` | a pending payment appears; bounded by `timeout` (default 10 s, caller-controllable) | no — exempt | | `GET /v1/lnurlp/tx/:id` | the payer's own broadcast reaches one confirmation (`tx` branch); 15 polls at 1 s. The `hex` branch broadcasts without awaiting confirmation, except on ICP, where it first waits for the payer's allowance (up to 3 attempts, 2 s apart) | no — exempt | | `GET /v1/node/:node/tx/:txId` | the transaction reaches one confirmation; bounded at 600 s | no — exempt | @@ -1228,7 +1303,7 @@ single DTO with two fields (PR #3772, 91 LOC, ~50% reduction). | Loading all then filtering in JS | SQL WHERE clause | | `any` type | Proper typed interface/class | | `string` for enum values | Typed enum | -| `@Interval(60000)` | `@DfxCron(CronExpression.EVERY_MINUTE)` | +| `@Interval(60000)` | `@DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER })` | | `eager: true` everywhere | Explicit relation loading | | Providing service in multiple modules | Single module, import from there | | `JSON.stringify(JSON.parse(...))` | Unnecessary — remove | diff --git a/docs/cron-jobs.md b/docs/cron-jobs.md index ccb451d1ed..4138c65b13 100644 --- a/docs/cron-jobs.md +++ b/docs/cron-jobs.md @@ -1,6 +1,6 @@ # Cron jobs -Every scheduled job this service runs: **131 `@DfxCron` declarations** across 92 files and 33 areas. +Every scheduled job this service runs: **140 `@DfxCron` declarations** across 98 files and 34 areas. ## Columns @@ -8,31 +8,58 @@ Every scheduled job this service runs: **131 `@DfxCron` declarations** across 92 | ------ | ------- | | **Interval** | The `CronExpression` / `CustomCronExpression` the job is registered with | | **Flag** | The `process:` kill switch that disables the job at runtime. `—` means the job has none and always runs | +| **Scope** | Which process registers the job: `worker`, `api`, or `both` | | **Job** | Class and method | | **File** | Path below `src/` | +## Scopes + +`scope` is a mandatory parameter of `@DfxCron` and says which process registers the job: +119 are `worker`, 5 are `api`, 16 are `both`. `CRON_ROLE` decides what a process is +(`worker`, `api`, or `all` for a single-process setup); a process runs its own scope plus `both`. + +`worker` is the normal case — anything writing to the database or driving business forward belongs +to exactly one process. `both` is for a job maintaining process-local state that a request path +also reads, so it must run everywhere; running it twice has to be harmless by construction, which +rules out database writes, mail and paid external calls. `api` is for state read only from a +request path — not for delivering to the connections a process holds open, because an `api` job +is leased and would run in one process while the connections are spread over all of them. + +Getting the scope wrong fails silently: the cache a job maintains simply stays empty in the +process that reads it. The rule that keeps that harmless is in CONTRIBUTING.md — a cache read in a +request path loads on demand, and a job may refresh it but must not be the only thing filling it. + ## Flags -110 of the 131 jobs carry a `process` flag, 21 do not. A job with a flag can be switched off +116 of the 140 jobs carry a `process` flag, 24 do not. A job with a flag can be switched off without a deploy — `DfxCronService` skips it when the process appears in the disabled set, which `ProcessService` refreshes from the `disabledProcesses` setting and the `DISABLED_PROCESSES` environment variable every 30 seconds. -A job **without** a flag runs unconditionally. That is deliberate for three of them — the -`ProcessService::resync*` jobs maintain the disabled set and the JWT denylists themselves, so -making them switchable would let a configuration change disable the mechanism that reads -configuration changes. For the remaining 18 it is simply an omission: +A job **without** a flag runs unconditionally. That is deliberate for nine of them. The four +`ProcessService::resync*` jobs maintain the disabled set, the JWT denylists and the staff +clearance allowlist themselves, so making them switchable would let a configuration change +disable the mechanism that reads configuration changes. `DfxCronService::reportRole` is the role +heartbeat: switched off, it would look exactly like a process that stopped reporting, which is +the condition it exists to make visible. `PaymentLinkGateway::checkConnections` drops websockets +that stopped answering, so switching it off would reinstate the unbounded growth it prevents. +`PaymentCronService::deliverPaymentUpdates` is the bridge between the process that writes a +payment and the one holding the connection: switched off it changes nothing in a single-process +setup, because `doSave` delivers directly there, and after the split it silently cuts delivery to +everything attached to the other container. `JwtRevocationSyncService::syncDeniedJwtAccounts` and +`StaffKycClearanceService::syncStaffKycClearance` fill the JWT denylist and the staff clearance +list from account state: switched off, neither empties its list — it stops writing it, so an +account blocked afterwards keeps its live access, and nothing reports the state. A switch whose +use is silent does not belong on a revocation path. For the remaining 15 it is simply an +omission: | Job | Interval | | --- | --- | -| `DexService::finalizePurchaseOrders` | 30 seconds | | `ExchangeController::checkTrades` | 30 seconds | | `AuthService::checkLists` | minute | -| `JwtRevocationSyncService::syncDeniedJwtAccounts` | minute | | `TransactionController::checkLists` | minute | | `UserDataService::processCleanupMailSecretCache` | minute | | `TransactionHelper::updateCache` | 5 minutes | -| `RefService::checkRefs` | hour | | `BuyService` / `SellService` / `SwapService` / `UserService` / `UserDataService` `::resetMonthlyVolumes` | 1st of month | | `BuyService` / `SellService` / `SwapService` / `UserService` / `UserDataService` `::resetAnnualVolumes` | year | @@ -44,11 +71,13 @@ New jobs should declare a flag unless there is a reason like the one above. | -------- | ---: | | second | 5 | | 10 seconds | 3 | -| 30 seconds | 8 | -| minute | 49 | -| 5 minutes | 17 | -| 10 minutes | 15 | +| 15 seconds | 1 | +| 30 seconds | 10 | +| minute | 52 | +| 5 minutes | 18 | +| 10 minutes | 16 | | hour | 16 | +| day at 3am | 1 | | day at 4am | 3 | | day at 5am | 1 | | day at 6am | 1 | @@ -62,24 +91,24 @@ Jobs by area: | Area | Jobs | Without flag | | ---- | ---: | -----------: | -| `subdomains/generic/user` | 15 | 7 | +| `subdomains/generic/user` | 16 | 7 | | `subdomains/core/monitoring` | 14 | — | | `subdomains/core/accounting` | 13 | — | | `subdomains/supporting/payin` | 12 | — | -| `integration/blockchain` | 6 | — | +| `integration/blockchain` | 7 | — | | `subdomains/core/buy-crypto` | 6 | 4 | +| `shared/services` | 5 | 5 | | `subdomains/core/sell-crypto` | 5 | 2 | -| `subdomains/core/payment-link` | 4 | — | +| `subdomains/supporting/payment` | 5 | 1 | +| `subdomains/core/payment-link` | 6 | 2 | | `subdomains/generic/kyc` | 4 | — | -| `subdomains/supporting/bank-tx` | 4 | — | | `subdomains/supporting/bank` | 4 | — | +| `subdomains/supporting/bank-tx` | 4 | — | | `subdomains/supporting/fiat-output` | 4 | — | | `subdomains/supporting/support-issue` | 4 | — | -| `shared` | 3 | 3 | | `subdomains/core/liquidity-management` | 3 | — | -| `subdomains/core/referral` | 3 | 1 | +| `subdomains/core/referral` | 3 | — | | `subdomains/core/trading` | 3 | — | -| `subdomains/supporting/payment` | 3 | 1 | | `subdomains/supporting/pricing` | 3 | — | | `integration/exchange` | 2 | 1 | | `subdomains/core/custody` | 2 | — | @@ -91,7 +120,8 @@ Jobs by area: | `subdomains/core/history` | 1 | 1 | | `subdomains/core/statistic` | 1 | — | | `subdomains/generic/admin` | 1 | — | -| `subdomains/supporting/dex` | 1 | 1 | +| `subdomains/supporting/dashboard` | 1 | — | +| `subdomains/supporting/dex` | 1 | — | | `subdomains/supporting/fiat-payin` | 1 | — | | `subdomains/supporting/notification` | 1 | — | | `subdomains/supporting/payout` | 1 | — | @@ -99,12 +129,16 @@ Jobs by area: ## How this list is produced Every `@DfxCron(` occurrence in `src/**/*.ts`. Decorator arguments are read by a balanced-paren -scan, so multi-line declarations are included — a line-based match misses four of them. The parsed -count is asserted against a raw text count of the decorator: **131 = 131**, no gap. Class and +scan, so multi-line declarations are included — a line-based match misses 27 of them. Interval, +flag and scope come from those arguments, so all three are as accurate as the source. The parsed +count is asserted against a raw text count of the decorator: **140 = 140**, no gap. Class and method come from the enclosing `export class` (including `export abstract class`) and the identifier following the decorator. -## Known discrepancy +## Known discrepancies + +Both come from the same place: this list counts **declarations**, while `DfxCronService` registers +what `DiscoveryService.getProviders()` hands it. The two are not the same set. `CitreaBaseStrategy::checkPayInEntries` is declared on an **abstract** class. NestJS discovers providers rather than classes, so such a job is registered once per concrete subclass, not once per @@ -112,138 +146,158 @@ declaration. There is currently one subclass, so the runtime count equals the de but a second subclass would silently add another registration of the same job, sharing the flag and the interval while running as an independent timer with its own lock. +`ExchangeController::checkTrades` is **never registered**. Its class is listed under `controllers:` +in `ExchangeModule` and nowhere under `providers:`, and `getProviders()` does not return +controllers — so the scan never sees the decorator. This predates the process split and is +unchanged by it; the job has never run. `TransactionController::checkLists` looks like the same +case but is not: `HistoryModule` lists that class under both `controllers:` and `providers:`, so +the job is registered — on the provider instance, which is a different object from the controller +instance the request handlers use. + +Resolving either one is a decision about the jobs, not about this inventory, so both are recorded +here rather than fixed in passing. Of the 140 declarations, 139 have a registration path. + ## Jobs -| Interval | Flag | Job | File | -| -------- | ---- | --- | ---- | -| second | `PAY_IN` | `BitcoinStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts` | -| second | `PAY_IN` | `FiroStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts` | -| second | `PAY_IN` | `MoneroStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts` | -| second | `MONITOR_CONNECTION_POOL` | `MonitorConnectionPoolService::monitorConnectionPool` | `subdomains/core/monitoring/monitor-connection-pool.service.ts` | -| second | `PAY_IN` | `ZanoStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts` | -| 10 seconds | `LIQUIDITY_MANAGEMENT` | `LiquidityManagementPipelineService::processPipelines` | `subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts` | -| 10 seconds | `MONITOR_CONNECTION_POOL` | `MonitorConnectionPoolService::monitorConnectionPoolStatic` | `subdomains/core/monitoring/monitor-connection-pool.service.ts` | -| 10 seconds | `MONITOR_EVENT_LOOP` | `MonitorEventLoopService::monitorEventLoop` | `subdomains/core/monitoring/monitor-event-loop.service.ts` | -| 30 seconds | `LNURL_AUTH_CACHE` | `AuthLnUrlService::processCleanupAccessToken` | `subdomains/generic/user/models/auth/auth-lnurl.service.ts` | -| 30 seconds | `BANK_TX` | `BankTxService::checkBankTx` | `subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts` | -| 30 seconds | — | `DexService::finalizePurchaseOrders` | `subdomains/supporting/dex/services/dex.service.ts` | -| 30 seconds | — | `ExchangeController::checkTrades` | `integration/exchange/controllers/exchange.controller.ts` | -| 30 seconds | `PAY_OUT` | `PayoutService::processOrders` | `subdomains/supporting/payout/services/payout.service.ts` | -| 30 seconds | — | `ProcessService::resyncDeniedJwtAccounts` | `shared/services/process.service.ts` | -| 30 seconds | — | `ProcessService::resyncDeniedJwtAddresses` | `shared/services/process.service.ts` | -| 30 seconds | — | `ProcessService::resyncDisabledProcesses` | `shared/services/process.service.ts` | -| minute | `PAY_OUT` | `AdminService::completeLiquidityOrders` | `subdomains/generic/admin/admin.service.ts` | -| minute | `MONITORING` | `AmlObserver::fetch` | `subdomains/core/monitoring/observers/aml.observer.ts` | -| minute | — | `AuthService::checkLists` | `subdomains/generic/user/models/auth/auth.service.ts` | -| minute | `BANK_DATA_VERIFICATION` | `BankDataService::checkAndSetActive` | `subdomains/generic/user/models/bank-data/bank-data.service.ts` | -| minute | `MONITORING` | `BankObserver::fetch` | `subdomains/core/monitoring/observers/bank.observer.ts` | -| minute | `BANK_TX_RETURN_MAIL` | `BankTxReturnNotificationService::sendBankTxReturnMail` | `subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts` | -| minute | `BUY_CRYPTO` | `BuyCryptoJobService::process` | `subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts` | -| minute | `BUY_FIAT` | `BuyFiatJobService::addFiatOutputs` | `subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts` | -| minute | `BUY_FIAT` | `BuyFiatJobService::checkCryptoPayIn` | `subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts` | -| minute | `BUY_FIAT_MAIL` | `BuyFiatNotificationService::sendNotificationMails` | `subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts` | -| minute | `PAY_IN` | `CardanoStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts` | -| minute | `MONITORING` | `CheckoutObserver::fetch` | `subdomains/core/monitoring/observers/checkout.observer.ts` | -| minute | `PAY_IN` | `CitreaBaseStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts` | -| minute | `CUSTODY` | `CustodyJobService::handleOrders` | `subdomains/core/custody/services/custody-job.service.ts` | -| minute | `FIAT_OUTPUT` | `FiatOutputJobService::fillFiatOutput` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | -| minute | `FIAT_PAY_IN` | `FiatPayInSyncService::syncCheckout` | `subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts` | -| minute | `PAY_IN` | `InternetComputerStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts` | -| minute | — | `JwtRevocationSyncService::syncDeniedJwtAccounts` | `subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts` | -| minute | `KYC` | `KycService::reviewKycSteps` | `subdomains/generic/kyc/services/kyc.service.ts` | -| minute | `LEDGER_BOOKING_BANK_TX` | `LedgerBookingJobService::runBankTx` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_BUY_CRYPTO` | `LedgerBookingJobService::runBuyCrypto` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_BUY_FIAT` | `LedgerBookingJobService::runBuyFiat` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_CRYPTO_INPUT` | `LedgerBookingJobService::runCryptoInput` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_EXCHANGE_TX` | `LedgerBookingJobService::runExchangeTx` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_LIQUIDITY_MANAGEMENT` | `LedgerBookingJobService::runLiquidityMgmt` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_LIQUIDITY_ORDER` | `LedgerBookingJobService::runLiquidityOrderDex` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_PAYOUT` | `LedgerBookingJobService::runPayoutOrder` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LEDGER_BOOKING_TRADING_ORDER` | `LedgerBookingJobService::runTradingOrder` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| minute | `LIQUIDITY_MANAGEMENT_CHECK_BALANCES` | `LiquidityManagementService::checkLiquidityBalances` | `subdomains/core/liquidity-management/services/liquidity-management.service.ts` | -| minute | `MONITORING` | `LiquidityObserver::fetch` | `subdomains/core/monitoring/observers/liquidity.observer.ts` | -| minute | `TRADING_LOG` | `LogJobService::saveTradingLog` | `subdomains/supporting/log/log-job.service.ts` | -| minute | `MONITORING` | `NodeHealthObserver::fetch` | `subdomains/core/monitoring/observers/node-health.observer.ts` | -| minute | `ORGANIZATION_SYNC` | `OrganizationService::syncOrganization` | `subdomains/generic/user/models/organization/organization.service.ts` | -| minute | `PAY_IN` | `PayInService::checkConfirmations` | `subdomains/supporting/payin/services/payin.service.ts` | -| minute | `PAY_IN` | `PayInService::forwardPayInEntries` | `subdomains/supporting/payin/services/payin.service.ts` | -| minute | `PAY_IN` | `PayInService::returnPayInEntries` | `subdomains/supporting/payin/services/payin.service.ts` | -| minute | `PAYMENT_CONFIRMATIONS` | `PaymentCronService::checkTxConfirmations` | `subdomains/core/payment-link/services/payment-cron.service.ts` | -| minute | `PAYMENT_EXPIRATION` | `PaymentCronService::processExpiredPayments` | `subdomains/core/payment-link/services/payment-cron.service.ts` | -| minute | `UPDATE_BLOCKCHAIN_FEE` | `PaymentLinkFeeService::updateFees` | `subdomains/core/payment-link/services/payment-link-fee.service.ts` | -| minute | `REALUNIT_QUOTE_COMPLETION` | `RealUnitJobService::completeSettledQuotes` | `subdomains/supporting/realunit/realunit-job.service.ts` | -| minute | `SUPPORT_BOT` | `SupportIssueJobService::sendAutoResponses` | `subdomains/supporting/support-issue/services/support-issue-job.service.ts` | -| minute | `TFA_CACHE` | `TfaService::processCleanupSecretCache` | `subdomains/generic/kyc/services/tfa.service.ts` | -| minute | `TRADING` | `TradingJobService::processOrders` | `subdomains/core/trading/services/trading-job.service.ts` | -| minute | `TRADING` | `TradingJobService::processRules` | `subdomains/core/trading/services/trading-job.service.ts` | -| minute | — | `TransactionController::checkLists` | `subdomains/core/history/controllers/transaction.controller.ts` | -| minute | `TX_MAIL` | `TransactionNotificationService::sendNotificationMails` | `subdomains/supporting/payment/services/transaction-notification.service.ts` | -| minute | `USER_DATA` | `UserDataJobService::fillUserData` | `subdomains/generic/user/models/user-data/user-data-job.service.ts` | -| minute | — | `UserDataService::processCleanupMailSecretCache` | `subdomains/generic/user/models/user-data/user-data.service.ts` | -| minute | `USER` | `UserJobService::fillUser` | `subdomains/generic/user/models/user/user-job.service.ts` | -| 5 minutes | `PRICING` | `AssetPricesJobService::updatePaymentPrices` | `subdomains/supporting/pricing/services/asset-prices-job.service.ts` | -| 5 minutes | `LNURL_AUTH_CACHE` | `AuthLnUrlService::processCleanupAuthCache` | `subdomains/generic/user/models/auth/auth-lnurl.service.ts` | -| 5 minutes | `BANK_TX_RETURN` | `BankTxReturnService::fillBankTxReturn` | `subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts` | -| 5 minutes | `BANK_TX` | `BankTxService::enrichYapealTransactions` | `subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts` | -| 5 minutes | `BLOCKCHAIN_CONFIG_CHECK` | `BlockchainConfigCheckService::logUnconfiguredClients` | `integration/blockchain/shared/services/blockchain-config-check.service.ts` | -| 5 minutes | `EXCHANGE_TX_SYNC` | `ExchangeTxService::syncExchangeJob` | `integration/exchange/services/exchange-tx.service.ts` | -| 5 minutes | `CRYPTO_PAYOUT` | `FaucetRequestService::checkFaucetRequests` | `subdomains/core/faucet-request/services/faucet-request.service.ts` | -| 5 minutes | `LEDGER_COA_BOOTSTRAP` | `LedgerBookingJobService::runCoaBootstrap` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | -| 5 minutes | `LEDGER_CUTOVER` | `LedgerCutoverService::run` | `subdomains/core/accounting/services/ledger-cutover.service.ts` | -| 5 minutes | `LIMIT_REQUEST_MAIL` | `LimitRequestNotificationService::sendNotificationMails` | `subdomains/supporting/support-issue/services/limit-request-notification.service.ts` | -| 5 minutes | `LIQUIDITY_MANAGEMENT` | `LiquidityManagementRuleService::reactivateRules` | `subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts` | -| 5 minutes | `PAY_IN_MAIL` | `PayInNotificationService::sendNotificationMails` | `subdomains/supporting/payin/services/payin-notification.service.ts` | -| 5 minutes | `REALUNIT_TRANSFER_RECONCILIATION` | `RealUnitJobService::reconcilePendingTransfers` | `subdomains/supporting/realunit/realunit-job.service.ts` | -| 5 minutes | `SUPPORT_BOT` | `SupportEscalationService::checkEscalations` | `subdomains/supporting/support-issue/services/support-escalation.service.ts` | -| 5 minutes | `TRADING` | `TradingJobService::reactivateRules` | `subdomains/core/trading/services/trading-job.service.ts` | -| 5 minutes | — | `TransactionHelper::updateCache` | `subdomains/supporting/payment/services/transaction-helper.ts` | -| 5 minutes | `WEBHOOK` | `WebhookNotificationService::sendWebhooks` | `subdomains/generic/user/services/webhook/webhook-notification.service.ts` | -| 10 minutes | `BANK_ACCOUNT` | `BankAccountService::reloadUncheckedBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | -| 10 minutes | `DEURO_LOG_INFO` | `DEuroService::processLogInfo` | `integration/blockchain/deuro/deuro.service.ts` | -| 10 minutes | `MONITORING` | `ExchangeObserver::fetch` | `subdomains/core/monitoring/observers/exchange.observer.ts` | -| 10 minutes | `MONITORING` | `ExternalServicesObserver::fetch` | `subdomains/core/monitoring/observers/external-services.observer.ts` | -| 10 minutes | `BLOCKCHAIN_FEE_UPDATE` | `FeeService::updateBlockchainFees` | `subdomains/supporting/payment/services/fee.service.ts` | -| 10 minutes | `FRANKENCOIN_LOG_INFO` | `FrankencoinService::processLogInfo` | `integration/blockchain/frankencoin/frankencoin.service.ts` | -| 10 minutes | `JUICE_LOG_INFO` | `JuiceService::processLogInfo` | `integration/blockchain/juice/juice.service.ts` | -| 10 minutes | `MONITORING` | `NodeBalanceObserver::fetch` | `subdomains/core/monitoring/observers/node-balance.observer.ts` | -| 10 minutes | `MAIL_RETRY` | `NotificationJobService::resendUncompletedMails` | `subdomains/supporting/notification/services/notification-job.service.ts` | -| 10 minutes | `PAY_IN` | `PayInService::updateFailedPayments` | `subdomains/supporting/payin/services/payin.service.ts` | -| 10 minutes | `MONITORING` | `PaymentObserver::fetch` | `subdomains/core/monitoring/observers/payment.observer.ts` | -| 10 minutes | `MONITORING` | `RealUnitW2wGasObserver::fetch` | `subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts` | -| 10 minutes | `REF_PAYOUT` | `RefRewardJobService::processPendingRefRewards` | `subdomains/core/referral/reward/services/ref-reward-job.service.ts` | -| 10 minutes | `MONITORING` | `UserObserver::fetch` | `subdomains/core/monitoring/observers/user.observer.ts` | -| 10 minutes | `ZANO_ASSET_WHITELIST` | `ZanoService::setupAssetWhitelist` | `integration/blockchain/zano/services/zano.service.ts` | -| hour | `PRICING` | `AssetPricesJobService::updatePrices` | `subdomains/supporting/pricing/services/asset-prices-job.service.ts` | -| hour | `BANK_ACCOUNT` | `BankAccountService::reloadErrorBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | -| hour | `BINANCE_PAY_CERTIFICATES_UPDATE` | `BinancePayService::updateCertificates` | `integration/binance-pay/services/binance-pay.service.ts` | -| hour | `BUY_CRYPTO_AGGREGATION` | `BuyCryptoJobService::checkAggregatingTransactions` | `subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts` | -| hour | `ASSET_DECIMALS` | `EvmDecimalsService::setDecimals` | `integration/blockchain/shared/evm/evm-decimals.service.ts` | -| hour | `FIAT_OUTPUT` | `FiatOutputFrickService::checkFrickOrderStatus` | `subdomains/supporting/fiat-output/fiat-output-frick.service.ts` | -| hour | `FIAT_OUTPUT` | `FiatOutputJobService::checkOlkypayOrderStatus` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | -| hour | `FIAT_OUTPUT` | `FiatOutputJobService::generateReports` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | -| hour | `PRICING` | `FiatPricesService::updatePrices` | `subdomains/supporting/pricing/services/fiat-prices.service.ts` | -| hour | `KYC_MAIL` | `KycNotificationService::sendNotificationMails` | `subdomains/generic/kyc/services/kyc-notification.service.ts` | -| hour | `PAYMENT_FORWARDING` | `PaymentCronService::forwardDeposits` | `subdomains/core/payment-link/services/payment-cron.service.ts` | -| hour | — | `RefService::checkRefs` | `subdomains/core/referral/process/ref.service.ts` | -| hour | `UPDATE_STATISTIC` | `StatisticService::doUpdate` | `subdomains/core/statistic/statistic.service.ts` | -| hour | `SUPPORT_BOT` | `SupportIssueJobService::autoOnHold` | `subdomains/supporting/support-issue/services/support-issue-job.service.ts` | -| hour | `BLACK_SQUAD_MAIL` | `UserDataNotificationService::sendNotificationMails` | `subdomains/generic/user/models/user-data/user-data-notification.service.ts` | -| hour | `VIRTUAL_IBAN_FRICK_ISSUANCE_RECONCILIATION` | `VirtualIbanFrickIssuanceReconciliationService::reconcileRetiredIssuanceReferences` | `subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts` | -| day at 4am | `CUSTODY` | `CustodyJobService::resetExpiredConfirmedOrders` | `subdomains/core/custody/services/custody-job.service.ts` | -| day at 4am | `KYC` | `KycService::checkIdentSteps` | `subdomains/generic/kyc/services/kyc.service.ts` | -| day at 4am | `LEDGER_MARK_TO_MARKET` | `LedgerMarkToMarketService::run` | `subdomains/core/accounting/services/ledger-mark-to-market.service.ts` | -| day at 5am | `LEDGER_RECONCILIATION` | `LedgerReconciliationService::run` | `subdomains/core/accounting/services/ledger-reconciliation.service.ts` | -| day at 6am | `REF_PAYOUT` | `RefRewardJobService::createPendingRefRewards` | `subdomains/core/referral/reward/services/ref-reward-job.service.ts` | -| day at 11pm | `LOG_CLEANUP` | `LogService::cleanup` | `subdomains/supporting/log/log.service.ts` | -| week | `BANK_ACCOUNT` | `BankAccountService::checkFailedBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | -| weekend | `SANCTION_SYNC` | `SanctionService::syncList` | `subdomains/core/aml/services/sanction.service.ts` | -| 1st day of month at midnight | — | `BuyService::resetMonthlyVolumes` | `subdomains/core/buy-crypto/routes/buy/buy.service.ts` | -| 1st day of month at midnight | — | `SellService::resetMonthlyVolumes` | `subdomains/core/sell-crypto/route/sell.service.ts` | -| 1st day of month at midnight | — | `SwapService::resetMonthlyVolumes` | `subdomains/core/buy-crypto/routes/swap/swap.service.ts` | -| 1st day of month at midnight | — | `UserDataService::resetMonthlyVolumes` | `subdomains/generic/user/models/user-data/user-data.service.ts` | -| 1st day of month at midnight | — | `UserService::resetMonthlyVolumes` | `subdomains/generic/user/models/user/user.service.ts` | -| year | — | `BuyService::resetAnnualVolumes` | `subdomains/core/buy-crypto/routes/buy/buy.service.ts` | -| year | — | `SellService::resetAnnualVolumes` | `subdomains/core/sell-crypto/route/sell.service.ts` | -| year | — | `SwapService::resetAnnualVolumes` | `subdomains/core/buy-crypto/routes/swap/swap.service.ts` | -| year | — | `UserDataService::resetAnnualVolumes` | `subdomains/generic/user/models/user-data/user-data.service.ts` | -| year | — | `UserService::resetAnnualVolumes` | `subdomains/generic/user/models/user/user.service.ts` | +| Interval | Flag | Scope | Job | File | +| -------- | ---- | ----- | --- | ---- | +| second | `PAY_IN` | `worker` | `BitcoinStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts` | +| second | `PAY_IN` | `worker` | `FiroStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts` | +| second | `PAY_IN` | `worker` | `MoneroStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts` | +| second | `MONITOR_CONNECTION_POOL` | `both` | `MonitorConnectionPoolService::monitorConnectionPool` | `subdomains/core/monitoring/monitor-connection-pool.service.ts` | +| second | `PAY_IN` | `worker` | `ZanoStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts` | +| 10 seconds | `LIQUIDITY_MANAGEMENT` | `worker` | `LiquidityManagementPipelineService::processPipelines` | `subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts` | +| 10 seconds | `MONITOR_CONNECTION_POOL` | `both` | `MonitorConnectionPoolService::monitorConnectionPoolStatic` | `subdomains/core/monitoring/monitor-connection-pool.service.ts` | +| 10 seconds | `MONITOR_EVENT_LOOP` | `both` | `MonitorEventLoopService::monitorEventLoop` | `subdomains/core/monitoring/monitor-event-loop.service.ts` | +| 15 seconds | — | `both` | `PaymentCronService::deliverPaymentUpdates` | `subdomains/core/payment-link/services/payment-cron.service.ts` | +| 30 seconds | `LNURL_AUTH_CACHE` | `both` | `AuthLnUrlService::processCleanupAccessToken` | `subdomains/generic/user/models/auth/auth-lnurl.service.ts` | +| 30 seconds | — | `both` | `PaymentLinkGateway::checkConnections` | `subdomains/core/payment-link/controllers/payment-link.gateway.ts` | +| 30 seconds | `BANK_TX` | `worker` | `BankTxService::checkBankTx` | `subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts` | +| 30 seconds | `DEX_PURCHASE_ORDER` | `worker` | `DexService::finalizePurchaseOrders` | `subdomains/supporting/dex/services/dex.service.ts` | +| 30 seconds | — | `api` | `ExchangeController::checkTrades` | `integration/exchange/controllers/exchange.controller.ts` | +| 30 seconds | `PAY_OUT` | `worker` | `PayoutService::processOrders` | `subdomains/supporting/payout/services/payout.service.ts` | +| 30 seconds | — | `both` | `ProcessService::resyncDeniedJwtAccounts` | `shared/services/process.service.ts` | +| 30 seconds | — | `both` | `ProcessService::resyncDeniedJwtAddresses` | `shared/services/process.service.ts` | +| 30 seconds | — | `both` | `ProcessService::resyncDisabledProcesses` | `shared/services/process.service.ts` | +| 30 seconds | — | `both` | `ProcessService::resyncStaffKycClearance` | `shared/services/process.service.ts` | +| minute | `PAY_OUT` | `worker` | `AdminService::completeLiquidityOrders` | `subdomains/generic/admin/admin.service.ts` | +| minute | `MONITORING` | `worker` | `AmlObserver::fetch` | `subdomains/core/monitoring/observers/aml.observer.ts` | +| minute | — | `both` | `AuthService::checkLists` | `subdomains/generic/user/models/auth/auth.service.ts` | +| minute | `BANK_DATA_VERIFICATION` | `worker` | `BankDataService::checkAndSetActive` | `subdomains/generic/user/models/bank-data/bank-data.service.ts` | +| minute | `MONITORING` | `worker` | `BankObserver::fetch` | `subdomains/core/monitoring/observers/bank.observer.ts` | +| minute | `BANK_TX_RETURN_MAIL` | `worker` | `BankTxReturnNotificationService::sendBankTxReturnMail` | `subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts` | +| minute | `BUY_CRYPTO` | `worker` | `BuyCryptoJobService::process` | `subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts` | +| minute | `BUY_FIAT` | `worker` | `BuyFiatJobService::addFiatOutputs` | `subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts` | +| minute | `BUY_FIAT` | `worker` | `BuyFiatJobService::checkCryptoPayIn` | `subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts` | +| minute | `BUY_FIAT_MAIL` | `worker` | `BuyFiatNotificationService::sendNotificationMails` | `subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts` | +| minute | `PAY_IN` | `worker` | `CardanoStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts` | +| minute | `MONITORING` | `worker` | `CheckoutObserver::fetch` | `subdomains/core/monitoring/observers/checkout.observer.ts` | +| minute | `PAY_IN` | `worker` | `CitreaBaseStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts` | +| minute | `CUSTODY` | `worker` | `CustodyJobService::handleOrders` | `subdomains/core/custody/services/custody-job.service.ts` | +| minute | `LATEST_BALANCE_CACHE` | `api` | `DashboardFinancialService::refreshLatestBalance` | `subdomains/supporting/dashboard/dashboard-financial.service.ts` | +| minute | `FIAT_OUTPUT` | `worker` | `FiatOutputJobService::fillFiatOutput` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | +| minute | `FIAT_PAY_IN` | `worker` | `FiatPayInSyncService::syncCheckout` | `subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts` | +| minute | `PAY_IN` | `worker` | `InternetComputerStrategy::checkPayInEntries` | `subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts` | +| minute | — | `worker` | `JwtRevocationSyncService::syncDeniedJwtAccounts` | `subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts` | +| minute | `KYC` | `worker` | `KycService::reviewKycSteps` | `subdomains/generic/kyc/services/kyc.service.ts` | +| minute | `LEDGER_BOOKING_BANK_TX` | `worker` | `LedgerBookingJobService::runBankTx` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_BUY_CRYPTO` | `worker` | `LedgerBookingJobService::runBuyCrypto` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_BUY_FIAT` | `worker` | `LedgerBookingJobService::runBuyFiat` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_CRYPTO_INPUT` | `worker` | `LedgerBookingJobService::runCryptoInput` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_EXCHANGE_TX` | `worker` | `LedgerBookingJobService::runExchangeTx` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_LIQUIDITY_MANAGEMENT` | `worker` | `LedgerBookingJobService::runLiquidityMgmt` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_LIQUIDITY_ORDER` | `worker` | `LedgerBookingJobService::runLiquidityOrderDex` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_PAYOUT` | `worker` | `LedgerBookingJobService::runPayoutOrder` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LEDGER_BOOKING_TRADING_ORDER` | `worker` | `LedgerBookingJobService::runTradingOrder` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| minute | `LIQUIDITY_MANAGEMENT_CHECK_BALANCES` | `worker` | `LiquidityManagementService::checkLiquidityBalances` | `subdomains/core/liquidity-management/services/liquidity-management.service.ts` | +| minute | `MONITORING` | `worker` | `LiquidityObserver::fetch` | `subdomains/core/monitoring/observers/liquidity.observer.ts` | +| minute | `TRADING_LOG` | `worker` | `LogJobService::saveTradingLog` | `subdomains/supporting/log/log-job.service.ts` | +| minute | `MONITORING` | `worker` | `NodeHealthObserver::fetch` | `subdomains/core/monitoring/observers/node-health.observer.ts` | +| minute | `ORGANIZATION_SYNC` | `worker` | `OrganizationService::syncOrganization` | `subdomains/generic/user/models/organization/organization.service.ts` | +| minute | `PAY_IN` | `worker` | `PayInService::checkConfirmations` | `subdomains/supporting/payin/services/payin.service.ts` | +| minute | `PAY_IN` | `worker` | `PayInService::forwardPayInEntries` | `subdomains/supporting/payin/services/payin.service.ts` | +| minute | `PAY_IN` | `worker` | `PayInService::returnPayInEntries` | `subdomains/supporting/payin/services/payin.service.ts` | +| minute | `PAYMENT_CONFIRMATIONS` | `worker` | `PaymentCronService::checkTxConfirmations` | `subdomains/core/payment-link/services/payment-cron.service.ts` | +| minute | `PAYMENT_EXPIRATION` | `worker` | `PaymentCronService::processExpiredPayments` | `subdomains/core/payment-link/services/payment-cron.service.ts` | +| minute | `UPDATE_BLOCKCHAIN_FEE` | `api` | `PaymentLinkFeeService::updateFees` | `subdomains/core/payment-link/services/payment-link-fee.service.ts` | +| minute | `REALUNIT_QUOTE_COMPLETION` | `worker` | `RealUnitJobService::completeSettledQuotes` | `subdomains/supporting/realunit/realunit-job.service.ts` | +| minute | — | `worker` | `StaffKycClearanceService::syncStaffKycClearance` | `subdomains/generic/user/models/user/staff-kyc-clearance.service.ts` | +| minute | `SUPPORT_BOT` | `worker` | `SupportIssueJobService::sendAutoResponses` | `subdomains/supporting/support-issue/services/support-issue-job.service.ts` | +| minute | `TFA_CACHE` | `both` | `TfaService::processCleanupSecretCache` | `subdomains/generic/kyc/services/tfa.service.ts` | +| minute | `TRADING` | `worker` | `TradingJobService::processOrders` | `subdomains/core/trading/services/trading-job.service.ts` | +| minute | `TRADING` | `worker` | `TradingJobService::processRules` | `subdomains/core/trading/services/trading-job.service.ts` | +| minute | — | `api` | `TransactionController::checkLists` | `subdomains/core/history/controllers/transaction.controller.ts` | +| minute | `TX_MAIL` | `worker` | `TransactionNotificationService::sendNotificationMails` | `subdomains/supporting/payment/services/transaction-notification.service.ts` | +| minute | `TX_REQUEST` | `worker` | `TransactionRequestService::txRequestStatusSync` | `subdomains/supporting/payment/services/transaction-request.service.ts` | +| minute | `USER_DATA` | `worker` | `UserDataJobService::fillUserData` | `subdomains/generic/user/models/user-data/user-data-job.service.ts` | +| minute | — | `both` | `UserDataService::processCleanupMailSecretCache` | `subdomains/generic/user/models/user-data/user-data.service.ts` | +| minute | `USER` | `worker` | `UserJobService::fillUser` | `subdomains/generic/user/models/user/user-job.service.ts` | +| 5 minutes | `PRICING` | `worker` | `AssetPricesJobService::updatePaymentPrices` | `subdomains/supporting/pricing/services/asset-prices-job.service.ts` | +| 5 minutes | `LNURL_AUTH_CACHE` | `both` | `AuthLnUrlService::processCleanupAuthCache` | `subdomains/generic/user/models/auth/auth-lnurl.service.ts` | +| 5 minutes | `BANK_TX_RETURN` | `worker` | `BankTxReturnService::fillBankTxReturn` | `subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts` | +| 5 minutes | `BANK_TX` | `worker` | `BankTxService::enrichYapealTransactions` | `subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts` | +| 5 minutes | `BLOCKCHAIN_CONFIG_CHECK` | `worker` | `BlockchainConfigCheckService::logUnconfiguredClients` | `integration/blockchain/shared/services/blockchain-config-check.service.ts` | +| 5 minutes | `EXCHANGE_TX_SYNC` | `worker` | `ExchangeTxService::syncExchangeJob` | `integration/exchange/services/exchange-tx.service.ts` | +| 5 minutes | `CRYPTO_PAYOUT` | `worker` | `FaucetRequestService::checkFaucetRequests` | `subdomains/core/faucet-request/services/faucet-request.service.ts` | +| 5 minutes | `LEDGER_COA_BOOTSTRAP` | `worker` | `LedgerBookingJobService::runCoaBootstrap` | `subdomains/core/accounting/services/ledger-booking-job.service.ts` | +| 5 minutes | `LEDGER_CUTOVER` | `worker` | `LedgerCutoverService::run` | `subdomains/core/accounting/services/ledger-cutover.service.ts` | +| 5 minutes | `LIMIT_REQUEST_MAIL` | `worker` | `LimitRequestNotificationService::sendNotificationMails` | `subdomains/supporting/support-issue/services/limit-request-notification.service.ts` | +| 5 minutes | `LIQUIDITY_MANAGEMENT` | `worker` | `LiquidityManagementRuleService::reactivateRules` | `subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts` | +| 5 minutes | `PAY_IN_MAIL` | `worker` | `PayInNotificationService::sendNotificationMails` | `subdomains/supporting/payin/services/payin-notification.service.ts` | +| 5 minutes | `REALUNIT_TRANSFER_RECONCILIATION` | `worker` | `RealUnitJobService::reconcilePendingTransfers` | `subdomains/supporting/realunit/realunit-job.service.ts` | +| 5 minutes | `SPARK_TOKEN_OPTIMIZATION` | `worker` | `SparkService::optimizeTokenOutputs` | `integration/blockchain/spark/spark.service.ts` | +| 5 minutes | `SUPPORT_BOT` | `worker` | `SupportEscalationService::checkEscalations` | `subdomains/supporting/support-issue/services/support-escalation.service.ts` | +| 5 minutes | `TRADING` | `worker` | `TradingJobService::reactivateRules` | `subdomains/core/trading/services/trading-job.service.ts` | +| 5 minutes | — | `both` | `TransactionHelper::updateCache` | `subdomains/supporting/payment/services/transaction-helper.ts` | +| 5 minutes | `WEBHOOK` | `worker` | `WebhookNotificationService::sendWebhooks` | `subdomains/generic/user/services/webhook/webhook-notification.service.ts` | +| 10 minutes | `BANK_ACCOUNT` | `worker` | `BankAccountService::reloadUncheckedBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | +| 10 minutes | `DEURO_LOG_INFO` | `worker` | `DEuroService::processLogInfo` | `integration/blockchain/deuro/deuro.service.ts` | +| 10 minutes | — | `both` | `DfxCronService::reportRole` | `shared/services/dfx-cron.service.ts` | +| 10 minutes | `MONITORING` | `worker` | `ExchangeObserver::fetch` | `subdomains/core/monitoring/observers/exchange.observer.ts` | +| 10 minutes | `MONITORING` | `worker` | `ExternalServicesObserver::fetch` | `subdomains/core/monitoring/observers/external-services.observer.ts` | +| 10 minutes | `BLOCKCHAIN_FEE_UPDATE` | `worker` | `FeeService::updateBlockchainFees` | `subdomains/supporting/payment/services/fee.service.ts` | +| 10 minutes | `FRANKENCOIN_LOG_INFO` | `worker` | `FrankencoinService::processLogInfo` | `integration/blockchain/frankencoin/frankencoin.service.ts` | +| 10 minutes | `JUICE_LOG_INFO` | `worker` | `JuiceService::processLogInfo` | `integration/blockchain/juice/juice.service.ts` | +| 10 minutes | `MONITORING` | `worker` | `NodeBalanceObserver::fetch` | `subdomains/core/monitoring/observers/node-balance.observer.ts` | +| 10 minutes | `MAIL_RETRY` | `worker` | `NotificationJobService::resendUncompletedMails` | `subdomains/supporting/notification/services/notification-job.service.ts` | +| 10 minutes | `PAY_IN` | `worker` | `PayInService::updateFailedPayments` | `subdomains/supporting/payin/services/payin.service.ts` | +| 10 minutes | `MONITORING` | `worker` | `PaymentObserver::fetch` | `subdomains/core/monitoring/observers/payment.observer.ts` | +| 10 minutes | `MONITORING` | `worker` | `RealUnitW2wGasObserver::fetch` | `subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts` | +| 10 minutes | `REF_PAYOUT` | `worker` | `RefRewardJobService::processPendingRefRewards` | `subdomains/core/referral/reward/services/ref-reward-job.service.ts` | +| 10 minutes | `MONITORING` | `worker` | `UserObserver::fetch` | `subdomains/core/monitoring/observers/user.observer.ts` | +| 10 minutes | `ZANO_ASSET_WHITELIST` | `worker` | `ZanoService::setupAssetWhitelist` | `integration/blockchain/zano/services/zano.service.ts` | +| hour | `PRICING` | `worker` | `AssetPricesJobService::updatePrices` | `subdomains/supporting/pricing/services/asset-prices-job.service.ts` | +| hour | `BANK_ACCOUNT` | `worker` | `BankAccountService::reloadErrorBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | +| hour | `BINANCE_PAY_CERTIFICATES_UPDATE` | `worker` | `BinancePayService::updateCertificates` | `integration/binance-pay/services/binance-pay.service.ts` | +| hour | `BUY_CRYPTO_AGGREGATION` | `worker` | `BuyCryptoJobService::checkAggregatingTransactions` | `subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts` | +| hour | `ASSET_DECIMALS` | `worker` | `EvmDecimalsService::setDecimals` | `integration/blockchain/shared/evm/evm-decimals.service.ts` | +| hour | `FIAT_OUTPUT` | `worker` | `FiatOutputFrickService::checkFrickOrderStatus` | `subdomains/supporting/fiat-output/fiat-output-frick.service.ts` | +| hour | `FIAT_OUTPUT` | `worker` | `FiatOutputJobService::checkOlkypayOrderStatus` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | +| hour | `FIAT_OUTPUT` | `worker` | `FiatOutputJobService::generateReports` | `subdomains/supporting/fiat-output/fiat-output-job.service.ts` | +| hour | `PRICING` | `worker` | `FiatPricesService::updatePrices` | `subdomains/supporting/pricing/services/fiat-prices.service.ts` | +| hour | `KYC_MAIL` | `worker` | `KycNotificationService::sendNotificationMails` | `subdomains/generic/kyc/services/kyc-notification.service.ts` | +| hour | `PAYMENT_FORWARDING` | `worker` | `PaymentCronService::forwardDeposits` | `subdomains/core/payment-link/services/payment-cron.service.ts` | +| hour | `REF_CLEANUP` | `worker` | `RefService::checkRefs` | `subdomains/core/referral/process/ref.service.ts` | +| hour | `UPDATE_STATISTIC` | `api` | `StatisticService::doUpdate` | `subdomains/core/statistic/statistic.service.ts` | +| hour | `SUPPORT_BOT` | `worker` | `SupportIssueJobService::autoOnHold` | `subdomains/supporting/support-issue/services/support-issue-job.service.ts` | +| hour | `BLACK_SQUAD_MAIL` | `worker` | `UserDataNotificationService::sendNotificationMails` | `subdomains/generic/user/models/user-data/user-data-notification.service.ts` | +| hour | `VIRTUAL_IBAN_FRICK_ISSUANCE_RECONCILIATION` | `worker` | `VirtualIbanFrickIssuanceReconciliationService::reconcileRetiredIssuanceReferences` | `subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts` | +| day at 3am | `TX_REQUEST_WAITING_EXPIRY` | `worker` | `TransactionRequestService::txRequestWaitingExpiryCheck` | `subdomains/supporting/payment/services/transaction-request.service.ts` | +| day at 4am | `CUSTODY` | `worker` | `CustodyJobService::resetExpiredConfirmedOrders` | `subdomains/core/custody/services/custody-job.service.ts` | +| day at 4am | `KYC` | `worker` | `KycService::checkIdentSteps` | `subdomains/generic/kyc/services/kyc.service.ts` | +| day at 4am | `LEDGER_MARK_TO_MARKET` | `worker` | `LedgerMarkToMarketService::run` | `subdomains/core/accounting/services/ledger-mark-to-market.service.ts` | +| day at 5am | `LEDGER_RECONCILIATION` | `worker` | `LedgerReconciliationService::run` | `subdomains/core/accounting/services/ledger-reconciliation.service.ts` | +| day at 6am | `REF_PAYOUT` | `worker` | `RefRewardJobService::createPendingRefRewards` | `subdomains/core/referral/reward/services/ref-reward-job.service.ts` | +| day at 11pm | `LOG_CLEANUP` | `worker` | `LogService::cleanup` | `subdomains/supporting/log/log.service.ts` | +| week | `BANK_ACCOUNT` | `worker` | `BankAccountService::checkFailedBankAccounts` | `subdomains/supporting/bank/bank-account/bank-account.service.ts` | +| weekend | `SANCTION_SYNC` | `worker` | `SanctionService::syncList` | `subdomains/core/aml/services/sanction.service.ts` | +| 1st day of month at midnight | — | `worker` | `BuyService::resetMonthlyVolumes` | `subdomains/core/buy-crypto/routes/buy/buy.service.ts` | +| 1st day of month at midnight | — | `worker` | `SellService::resetMonthlyVolumes` | `subdomains/core/sell-crypto/route/sell.service.ts` | +| 1st day of month at midnight | — | `worker` | `SwapService::resetMonthlyVolumes` | `subdomains/core/buy-crypto/routes/swap/swap.service.ts` | +| 1st day of month at midnight | — | `worker` | `UserDataService::resetMonthlyVolumes` | `subdomains/generic/user/models/user-data/user-data.service.ts` | +| 1st day of month at midnight | — | `worker` | `UserService::resetMonthlyVolumes` | `subdomains/generic/user/models/user/user.service.ts` | +| year | — | `worker` | `BuyService::resetAnnualVolumes` | `subdomains/core/buy-crypto/routes/buy/buy.service.ts` | +| year | — | `worker` | `SellService::resetAnnualVolumes` | `subdomains/core/sell-crypto/route/sell.service.ts` | +| year | — | `worker` | `SwapService::resetAnnualVolumes` | `subdomains/core/buy-crypto/routes/swap/swap.service.ts` | +| year | — | `worker` | `UserDataService::resetAnnualVolumes` | `subdomains/generic/user/models/user-data/user-data.service.ts` | +| year | — | `worker` | `UserService::resetAnnualVolumes` | `subdomains/generic/user/models/user/user.service.ts` | diff --git a/migration/1785600000000-AddCronLease.js b/migration/1785600000000-AddCronLease.js new file mode 100644 index 0000000000..46841cb3d2 --- /dev/null +++ b/migration/1785600000000-AddCronLease.js @@ -0,0 +1,62 @@ +/** + * @typedef {import('typeorm').MigrationInterface} MigrationInterface + * @typedef {import('typeorm').QueryRunner} QueryRunner + */ + +/** + * Cross-process lease for scheduled jobs. + * + * Until now the only thing stopping a job from running twice was `LockClass`, and its state is a + * field in process memory — it cannot see a second process. That was acceptable while the API ran + * as a single process. With the HTTP process and the worker split apart, "exactly one process runs + * this job" became an assumption held up by configuration, a runbook sentence and an alert — and + * that alert reports a WRONG ROLE, not a double run: it reads the role each process states, which + * says what a process WOULD run, never what two of them did. For a path that moves money, an + * assumption checked from the outside is the second-best answer. + * + * This table is what a second process has to get past before it may start such a job: a job scoped + * to exactly one process must take a row here before it starts, and the row is claimable by one + * process at a time until it expires. The expiry is why this is not an exclusion — if the + * holder can no longer renew, a second process can claim the row while the first is still working, + * and how long the two then overlap is not bounded by anything here. See CronLeaseService, "What it + * does not do". + * + * No foreign keys, deliberately: the table is infrastructure, not domain data, and a key into a + * domain table would tie a coordination row to a schema it has no business depending on. + * `name` is the primary key, so the claim is a single atomic upsert with no index to keep in sync. + * + * The primary key carries the name TypeORM derives for it, per the rule in CONTRIBUTING.md: `PK_` + * plus the first 27 characters of sha1('cron_lease_name'). A hand-picked name would not be + * recognised as its own by a schema comparison, which would offer to drop and recreate it. + * + * Both timestamps carry their time zone. They are compared against `now()` inside the claim and + * renewal statements, and a value without a zone on one side of that comparison is resolved + * through whatever time zone the session carries — which makes the same row expire an hour late + * or an hour early across a daylight saving change, and inconsistently between two sessions that + * disagree. An hour late is a job that runs nowhere, an hour early is two processes running it at + * once. + * + * The shape of the table is mirrored by src/shared/models/cron-lease/cron-lease.entity.ts, without + * which a generated migration would read the table as one to drop. + * + * @class @implements {MigrationInterface} + */ +module.exports = class AddCronLease1785600000000 { + name = 'AddCronLease1785600000000'; + + /** + * @param {QueryRunner} queryRunner + */ + async up(queryRunner) { + await queryRunner.query( + `CREATE TABLE "cron_lease" ("name" character varying(256) NOT NULL, "owner" character varying(256) NOT NULL, "acquired" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(), "expires" TIMESTAMP WITH TIME ZONE NOT NULL, CONSTRAINT "PK_a12c181c2b26f33be13d55a15af" PRIMARY KEY ("name"))`, + ); + } + + /** + * @param {QueryRunner} queryRunner + */ + async down(queryRunner) { + await queryRunner.query(`DROP TABLE "cron_lease"`); + } +}; diff --git a/migration/1785620000000-AddPaymentLinkPaymentDeviceIdIndex.js b/migration/1785620000000-AddPaymentLinkPaymentDeviceIdIndex.js new file mode 100644 index 0000000000..9d1cccfa91 --- /dev/null +++ b/migration/1785620000000-AddPaymentLinkPaymentDeviceIdIndex.js @@ -0,0 +1,69 @@ +/** + * @typedef {import('typeorm').MigrationInterface} MigrationInterface + * @typedef {import('typeorm').QueryRunner} QueryRunner + */ + +/** + * Index `payment_link_payment ("deviceId")`. + * + * `PaymentLinkPaymentService.deliverToConnectedDevices` introduces the first lookup by that + * column: while this process holds a websocket connection open for a device, it asks every 15 + * seconds whether a payment of that device has reached a state the device has to be told about. + * `deviceId` carried no index, so that lookup would scan the whole table on every one of those + * ticks for as long as a device stays connected. + * + * A single-column index is enough for the shape of the query. It filters `deviceId IN (…)` on the + * handful of devices connected to this process; `expiryDate` and the status conditions are applied + * to what that yields and are not part of the index. What the index removes is the scan of every + * OTHER device's payments, which is the part that grows with the table. What it leaves is one + * device's own history — bounded by how much that terminal has taken, not by the query. + * + * The name is the deterministic one TypeORM's `DefaultNamingStrategy` derives, since CONTRIBUTING + * disallows custom index names: `IDX_` followed by the first 26 hex characters of + * `sha1('payment_link_payment_deviceId')` (table name + `_` + the column name). It is pinned + * against the entity in `payment-link-payment.entity.spec.ts`, so a rename on either side fails a + * test rather than producing a second index the next generated migration would add. + * + * `CREATE INDEX CONCURRENTLY` is not used: migrations run inside a transaction (`migrationsRun` in + * `src/config/config.ts`, TypeORM's default `migrationsTransactionMode: 'all'`), and CONCURRENTLY + * is not allowed there. The plain form takes a SHARE lock, which blocks writes to the table — and + * because locks are released at COMMIT, it holds until the whole pending batch commits. + * + * @class + * @implements {MigrationInterface} + */ +module.exports = class AddPaymentLinkPaymentDeviceIdIndex1785620000000 { + name = 'AddPaymentLinkPaymentDeviceIdIndex1785620000000'; + + /** + * @param {QueryRunner} queryRunner + */ + async up(queryRunner) { + // SET LOCAL is scoped to the whole TRANSACTION, and under `migrationsTransactionMode: 'all'` + // that transaction is the entire pending batch — so this stays in force for every migration + // that runs after it in the same deployment, not only for the statement below. That is + // deliberate but worth knowing: a later migration that must wait on a lock inherits the five + // seconds and fails the whole release rather than waiting. Whoever adds one sets its own + // value. + // + // It bounds the WAIT for the lock, not how long the lock is held. + await queryRunner.query(`SET LOCAL lock_timeout = '5s'`); + await queryRunner.query(`CREATE INDEX "IDX_8a9b97a10b3db9c64d45ae4d38" ON "payment_link_payment" ("deviceId")`); + } + + /** + * @param {QueryRunner} queryRunner + */ + async down(queryRunner) { + // SET LOCAL is scoped to the whole TRANSACTION, and under `migrationsTransactionMode: 'all'` + // that transaction is the entire pending batch — so this stays in force for every migration + // that runs after it in the same deployment, not only for the statement below. That is + // deliberate but worth knowing: a later migration that must wait on a lock inherits the five + // seconds and fails the whole release rather than waiting. Whoever adds one sets its own + // value. + // + // It bounds the WAIT for the lock, not how long the lock is held. + await queryRunner.query(`SET LOCAL lock_timeout = '5s'`); + await queryRunner.query(`DROP INDEX "public"."IDX_8a9b97a10b3db9c64d45ae4d38"`); + } +}; diff --git a/package-lock.json b/package-lock.json index b293fef6e1..c46015f352 100644 --- a/package-lock.json +++ b/package-lock.json @@ -44,7 +44,9 @@ "@noble/curves": "^1.9.7", "@opentelemetry/api": "^1.9.1", "@opentelemetry/auto-instrumentations-node": "^0.76.0", + "@opentelemetry/exporter-metrics-otlp-http": "^0.218.0", "@opentelemetry/exporter-trace-otlp-http": "^0.218.0", + "@opentelemetry/sdk-metrics": "2.7.1", "@opentelemetry/sdk-node": "^0.218.0", "@opentelemetry/sdk-trace-base": "^2.7.1", "@railgun-community/engine": "^9.4.0", diff --git a/package.json b/package.json index c47a2bed43..7f1253f473 100644 --- a/package.json +++ b/package.json @@ -64,7 +64,9 @@ "@noble/curves": "^1.9.7", "@opentelemetry/api": "^1.9.1", "@opentelemetry/auto-instrumentations-node": "^0.76.0", + "@opentelemetry/exporter-metrics-otlp-http": "^0.218.0", "@opentelemetry/exporter-trace-otlp-http": "^0.218.0", + "@opentelemetry/sdk-metrics": "2.7.1", "@opentelemetry/sdk-node": "^0.218.0", "@opentelemetry/sdk-trace-base": "^2.7.1", "@railgun-community/engine": "^9.4.0", diff --git a/src/__tests__/migration-constraint-naming.spec.ts b/src/__tests__/migration-constraint-naming.spec.ts new file mode 100644 index 0000000000..a4518326e1 --- /dev/null +++ b/src/__tests__/migration-constraint-naming.spec.ts @@ -0,0 +1,84 @@ +import { createHash } from 'crypto'; +import { readFileSync, readdirSync } from 'fs'; +import { join } from 'path'; + +/** + * Hand-written schema migrations must name their constraints the way TypeORM does. CONTRIBUTING.md + * states the rule and the algorithm; this checks that the files obey it. + * + * It is not a style question. A constraint TypeORM does not recognise as its own is one it offers + * to create — a schema comparison sees a name it would never have produced, reports the constraint + * as missing, and a generated migration or a `synchronize` run acts on that. + * + * Two checks, deliberately: the recomputation covers the primary keys declared inside a + * `CREATE TABLE`, where table and columns are both in front of us, and the shape check covers + * every other constraint kind, where they are not. The shape check is the weaker statement — a + * hexadecimal name can still be the wrong hexadecimal name — but it catches the case this guard + * was written for, a name assembled out of words. + */ +const MIGRATIONS = join(__dirname, '..', '..', 'migration'); + +/** From CONTRIBUTING.md: `_ + sha1(tableName + '_' + columnNames.sort().join('_'))`. */ +function typeormName(prefix: string, length: number, table: string, columns: string[]): string { + return `${prefix}_${createHash('sha1') + .update(`${table}_${[...columns].sort().join('_')}`) + .digest('hex') + .substring(0, length)}`; +} + +interface Migration { + file: string; + content: string; +} + +describe('migration constraint naming', () => { + const migrations: Migration[] = readdirSync(MIGRATIONS) + .filter((file) => file.endsWith('.js')) + .map((file) => ({ file, content: readFileSync(join(MIGRATIONS, file), 'utf8') })); + + it('finds the migrations to check', () => { + // Guards against the whole suite passing because the directory was read from the wrong place. + expect(migrations.length).toBeGreaterThan(50); + }); + + describe('primary keys declared in a CREATE TABLE', () => { + const declared = migrations.flatMap(({ file, content }) => + [...content.matchAll(/CREATE TABLE "(\w+)" \((?:.*?)CONSTRAINT "(PK_\w+)" PRIMARY KEY \(([^)]*)\)/gs)].map( + ([, table, name, columns]) => ({ + file, + table, + name, + columns: columns.split(',').map((column) => column.trim().replace(/"/g, '')), + }), + ), + ); + + it('finds primary keys to check', () => { + expect(declared.length).toBeGreaterThan(50); + }); + + it('names every one of them the way TypeORM would', () => { + const wrong = declared + .filter(({ table, name, columns }) => name !== typeormName('PK', 27, table, columns)) + .map(({ file, table, name, columns }) => ({ + file, + name, + expected: typeormName('PK', 27, table, columns), + })); + + expect(wrong).toEqual([]); + }); + }); + + it('gives every constraint a hashed name rather than a spelled-out one', () => { + // The shape the algorithm produces: a prefix and hexadecimal. A name built from the table and + // column it belongs to reads correctly and is exactly the case this catches. + const spelledOut = migrations.flatMap(({ file, content }) => + [...content.matchAll(/CONSTRAINT "((?:PK|FK|UQ|DF|REL|IDX|CHK)_\w+)"/g)] + .map(([, name]) => ({ file, name })) + .filter(({ name }) => !/^(?:PK|FK|UQ|DF|REL|IDX|CHK)_[a-f0-9]+$/.test(name)), + ); + + expect(spelledOut).toEqual([]); + }); +}); diff --git a/src/__tests__/runtime-metrics.spec.ts b/src/__tests__/runtime-metrics.spec.ts new file mode 100644 index 0000000000..583a629fdc --- /dev/null +++ b/src/__tests__/runtime-metrics.spec.ts @@ -0,0 +1,55 @@ +import { IntervalHistogram } from 'perf_hooks'; +import { toEventLoopSample } from '../runtime-metrics'; + +const NS_PER_S = 1e9; + +function fakeHistogram( + values: Partial> & { count: number }, +): IntervalHistogram { + return { + ...values, + percentile: (p: number) => { + const byPercentile: Record = { + 50: 0.05 * NS_PER_S, + 90: 0.5 * NS_PER_S, + 99: 2 * NS_PER_S, + }; + + return byPercentile[p]; + }, + } as unknown as IntervalHistogram; +} + +describe('toEventLoopSample', () => { + it('converts histogram nanoseconds to seconds', () => { + const histogram = fakeHistogram({ + count: 100, + min: 0.001 * NS_PER_S, + max: 5 * NS_PER_S, + mean: 0.3 * NS_PER_S, + }); + + const sample = toEventLoopSample(histogram, 0.85); + + expect(sample.utilization).toBe(0.85); + expect(sample.delay).toEqual({ min: 0.001, max: 5, mean: 0.3, p50: 0.05, p90: 0.5, p99: 2 }); + }); + + it('reports zeros for an empty histogram instead of the Infinity/0 sentinels Node returns', () => { + // A freshly reset histogram returns min = Infinity and max = 0. Exporting Infinity would + // break the series for every consumer, so an empty window must read as all zeros. + const histogram = fakeHistogram({ count: 0, min: Infinity, max: 0, mean: NaN }); + + const sample = toEventLoopSample(histogram, 0); + + expect(sample.delay).toEqual({ min: 0, max: 0, mean: 0, p50: 0, p90: 0, p99: 0 }); + expect(Object.values(sample.delay).every(Number.isFinite)).toBe(true); + }); + + it('passes utilization through unchanged as a 0..1 ratio', () => { + const histogram = fakeHistogram({ count: 1, min: 0, max: 0, mean: 0 }); + + expect(toEventLoopSample(histogram, 0).utilization).toBe(0); + expect(toEventLoopSample(histogram, 1).utilization).toBe(1); + }); +}); diff --git a/src/__tests__/tracing.spec.ts b/src/__tests__/tracing.spec.ts index 424a36e058..7c984cbcb0 100644 --- a/src/__tests__/tracing.spec.ts +++ b/src/__tests__/tracing.spec.ts @@ -9,10 +9,16 @@ jest.mock('@opentelemetry/auto-instrumentations-node', () => ({ jest.mock('@opentelemetry/exporter-trace-otlp-http', () => ({ OTLPTraceExporter: jest.fn(), })); +jest.mock('@opentelemetry/exporter-metrics-otlp-http', () => ({ + OTLPMetricExporter: jest.fn(), +})); +jest.mock('@opentelemetry/sdk-metrics', () => ({ + PeriodicExportingMetricReader: jest.fn(), +})); import { SpanKind, SpanStatusCode } from '@opentelemetry/api'; import { ReadableSpan } from '@opentelemetry/sdk-trace-base'; -import { ClientErrorSpanProcessor, isClientError, startTracing } from '../tracing'; +import { ClientErrorSpanProcessor, isClientError, startTracing, tracingServiceName } from '../tracing'; function fakeSpan(kind: SpanKind, statusCode: SpanStatusCode, httpStatus?: number): ReadableSpan { return { @@ -89,3 +95,26 @@ describe('startTracing', () => { expect(mockStart).toHaveBeenCalledTimes(1); }); }); + +describe('tracingServiceName', () => { + const original = process.env.CRON_ROLE; + + afterEach(() => { + if (original === undefined) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = original; + }); + + it('reports the worker under its own name', () => { + // Both processes run the same image and would otherwise report as one service, leaving a + // consumer of the traces unable to tell them apart. + process.env.CRON_ROLE = 'worker'; + expect(tracingServiceName()).toBe('dfx-api-worker'); + }); + + it.each(['api', 'all', undefined])('reports %p as the API service', (role) => { + if (role === undefined) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = role; + + expect(tracingServiceName()).toBe('dfx-api'); + }); +}); diff --git a/src/config/__tests__/cron-role.config.spec.ts b/src/config/__tests__/cron-role.config.spec.ts new file mode 100644 index 0000000000..eaebf978b7 --- /dev/null +++ b/src/config/__tests__/cron-role.config.spec.ts @@ -0,0 +1,62 @@ +import { Config, ConfigService, CronRole, GetConfig, parseCronRole } from '../config'; + +describe('parseCronRole', () => { + it('accepts the three roles', () => { + expect(parseCronRole('all')).toBe(CronRole.ALL); + expect(parseCronRole('api')).toBe(CronRole.API); + expect(parseCronRole('worker')).toBe(CronRole.WORKER); + }); + + it.each([undefined, '', ' ', 'All', 'WORKER', 'api ', 'true', 'none'])( + 'throws on %p instead of picking a role', + (value) => { + // Verifies that a missing, empty or unknown value is rejected rather than mapped to a + // default: `parseCronRole` has no fallback branch, and the empty string takes the same path + // as any other invalid value. + expect(() => parseCronRole(value)).toThrow(/expected one of all, api, worker/); + }, + ); + + it('does not accept a scope value as a role', () => { + // `both` is a property of a job, not an operating mode of a process. Accepting it here would + // blur the two axes the split depends on. + expect(() => parseCronRole('both')).toThrow(); + }); +}); + +describe('Config.cronRole', () => { + const original = process.env.CRON_ROLE; + + afterEach(() => { + if (original == null) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = original; + + new ConfigService(GetConfig()); + }); + + // Covers the wiring env -> parseCronRole -> Config that DfxCronService reads, which + // unit-testing the parser alone would leave unverified. + it.each([ + ['all', CronRole.ALL], + ['api', CronRole.API], + ['worker', CronRole.WORKER], + ])('maps CRON_ROLE=%s to %s', (value, expected) => { + process.env.CRON_ROLE = value; + + new ConfigService(GetConfig()); + + expect(Config.cronRole).toBe(expected); + }); + + it('refuses to build a configuration without a role', () => { + delete process.env.CRON_ROLE; + + expect(() => GetConfig()).toThrow(/expected one of all, api, worker/); + }); + + it('refuses to build a configuration from an invalid value', () => { + process.env.CRON_ROLE = 'wroker'; + + expect(() => GetConfig()).toThrow(/expected one of all, api, worker/); + }); +}); diff --git a/src/config/config.ts b/src/config/config.ts index 785a112b1d..7c142ced99 100644 --- a/src/config/config.ts +++ b/src/config/config.ts @@ -36,6 +36,22 @@ export enum Environment { PRD = 'prd', } +/** + * Operating mode of this process, read from CRON_ROLE. + * + * `scope` on a job describes a property of the job, this describes the process running it. + * Keeping the two apart is what allows the same image to run as an HTTP process and as a + * background worker without either knowing about the other. + */ +export enum CronRole { + /** One process runs everything: local development, tests, and any deployment without a worker. */ + ALL = 'all', + /** Serves HTTP; runs only jobs scoped `api` or `both`. */ + API = 'api', + /** Runs the background work; only jobs scoped `worker` or `both`. */ + WORKER = 'worker', +} + export type StorageWriteMode = 'azure' | 'dual' | 's3'; export type StorageReadSource = 'azure' | 's3'; @@ -1496,6 +1512,16 @@ export class Configuration { return splitWithdrawKeys(process.env.EVM_WALLETS); } + // Background jobs and HTTP requests share a single Node event loop, so a busy scheduler + // delays every incoming request on the same process. The role decides which jobs this + // process registers, which is what allows running the same image twice: once serving HTTP, + // once running the background work. + // + // Note this is deliberately independent of DISABLED_PROCESSES, which only skips jobs that + // declare a `process` — jobs without one would keep running and, in a second process, + // run twice. Cron locks are per-process and do not guard across processes. + cronRole = parseCronRole(process.env.CRON_ROLE); + // --- HELPERS --- // disabledProcesses = () => process.env.DISABLED_PROCESSES === '*' @@ -1503,6 +1529,28 @@ export class Configuration { : ((process.env.DISABLED_PROCESSES?.split(',') ?? []) as Process[]); } +/** + * Reads CRON_ROLE, the operating mode of this process. + * + * There is no default, and a missing or unknown value aborts the boot. Every possible default + * lets a misconfiguration run silently: defaulting to `worker` would make a misconfigured API + * process run all background work a second time, defaulting to `api` would make a misconfigured + * worker do nothing at all. Neither produces an error, and duplicate execution of financial + * jobs is far more damaging than a failed boot. + * + * The empty string is rejected for the same reason: a `CRON_ROLE=` line in an env file or an + * unresolved `${VAR}` both arrive here as one. + * + * `all` is not a convenience value but the single-process mode: one process runs every job, + * which is what local development, the test suite and any environment without a separate worker + * need. + */ +export function parseCronRole(value?: string): CronRole { + if (value != null && (Object.values(CronRole) as string[]).includes(value)) return value as CronRole; + + throw new Error(`Invalid CRON_ROLE value '${value ?? ''}': expected one of ${Object.values(CronRole).join(', ')}`); +} + function readCert(): string | undefined { const path = process.env.LIGHTNING_API_CERTIFICATE_PATH; if (path) { diff --git a/src/integration/binance-pay/services/binance-pay.service.ts b/src/integration/binance-pay/services/binance-pay.service.ts index 9de4b396e2..5ddee4eea2 100644 --- a/src/integration/binance-pay/services/binance-pay.service.ts +++ b/src/integration/binance-pay/services/binance-pay.service.ts @@ -5,7 +5,7 @@ import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { HttpService } from 'src/shared/services/http.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { TransferInfo } from 'src/subdomains/core/payment-link/dto/payment-link.dto'; import { PaymentLinkPayment } from 'src/subdomains/core/payment-link/entities/payment-link-payment.entity'; @@ -225,7 +225,7 @@ export class BinancePayService implements C2BPaymentLinkProvider { try { const headers = this.getHeaders({}); diff --git a/src/integration/blockchain/deuro/deuro.service.ts b/src/integration/blockchain/deuro/deuro.service.ts index 002ca71236..e510138b56 100644 --- a/src/integration/blockchain/deuro/deuro.service.ts +++ b/src/integration/blockchain/deuro/deuro.service.ts @@ -5,7 +5,7 @@ import { Contract } from 'ethers'; import { Config } from 'src/config/config'; import { Asset } from 'src/shared/models/asset/asset.entity'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CreateLogDto } from 'src/subdomains/supporting/log/dto/create-log.dto'; import { LogSeverity } from 'src/subdomains/supporting/log/log.entity'; @@ -60,7 +60,7 @@ export class DEuroService extends FrankencoinBasedService implements OnModuleIni this.deuroClient = new DEuroClient(this.getEvmClient()); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.DEURO_LOG_INFO }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.DEURO_LOG_INFO }) async processLogInfo(): Promise { if (!Config.blockchain.deuro.graphUrl || !Config.blockchain.deuro.apiUrl) { this.logger.warn('DEuro graphUrl/apiUrl not configured - skipping processLogInfo'); diff --git a/src/integration/blockchain/frankencoin/frankencoin.service.ts b/src/integration/blockchain/frankencoin/frankencoin.service.ts index 468dd42899..08a38d8e3a 100644 --- a/src/integration/blockchain/frankencoin/frankencoin.service.ts +++ b/src/integration/blockchain/frankencoin/frankencoin.service.ts @@ -4,7 +4,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Contract } from 'ethers'; import { Config } from 'src/config/config'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { CreateLogDto } from 'src/subdomains/supporting/log/dto/create-log.dto'; import { LogSeverity } from 'src/subdomains/supporting/log/log.entity'; import { LogService } from 'src/subdomains/supporting/log/log.service'; @@ -51,7 +51,7 @@ export class FrankencoinService extends FrankencoinBasedService implements OnMod this.frankencoinClient = new FrankencoinClient(this.getEvmClient()); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.FRANKENCOIN_LOG_INFO }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.FRANKENCOIN_LOG_INFO }) async processLogInfo() { if (!Config.blockchain.frankencoin.contractAddress.xchf) { this.logger.warn('Frankencoin xchf contract not configured - skipping processLogInfo'); diff --git a/src/integration/blockchain/juice/juice.service.ts b/src/integration/blockchain/juice/juice.service.ts index 0b6e499db5..dc4504270e 100644 --- a/src/integration/blockchain/juice/juice.service.ts +++ b/src/integration/blockchain/juice/juice.service.ts @@ -5,7 +5,7 @@ import { Contract } from 'ethers'; import { Config } from 'src/config/config'; import { Asset } from 'src/shared/models/asset/asset.entity'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CreateLogDto } from 'src/subdomains/supporting/log/dto/create-log.dto'; import { LogSeverity } from 'src/subdomains/supporting/log/log.entity'; @@ -61,7 +61,7 @@ export class JuiceService extends FrankencoinBasedService implements OnModuleIni return this.registryService.getClient(Blockchain.CITREA) as EvmClient; } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.JUICE_LOG_INFO }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.JUICE_LOG_INFO }) async processLogInfo(): Promise { if (!Config.blockchain.juice.graphUrl || !Config.blockchain.juice.apiUrl) { this.logger.warn('Juice graphUrl/apiUrl not configured - skipping processLogInfo'); diff --git a/src/integration/blockchain/shared/evm/evm-decimals.service.ts b/src/integration/blockchain/shared/evm/evm-decimals.service.ts index 19ac56f539..cb92762ebf 100644 --- a/src/integration/blockchain/shared/evm/evm-decimals.service.ts +++ b/src/integration/blockchain/shared/evm/evm-decimals.service.ts @@ -5,7 +5,7 @@ import { AssetService } from 'src/shared/models/asset/asset.service'; import { UpdateResult } from 'src/shared/models/entity'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { BlockchainRegistryService } from '../services/blockchain-registry.service'; import { EvmBlockchains } from '../util/blockchain.util'; @@ -19,7 +19,7 @@ export class EvmDecimalsService { ) {} // --- JOBS --- // - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.ASSET_DECIMALS, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.ASSET_DECIMALS, timeout: 1800 }) async setDecimals() { const assets = await this.assetService.getEvmAssetsWithoutDecimals(EvmBlockchains); diff --git a/src/integration/blockchain/shared/services/blockchain-config-check.service.ts b/src/integration/blockchain/shared/services/blockchain-config-check.service.ts index 5341e72c0f..9a163ee51c 100644 --- a/src/integration/blockchain/shared/services/blockchain-config-check.service.ts +++ b/src/integration/blockchain/shared/services/blockchain-config-check.service.ts @@ -7,7 +7,7 @@ import { BlockchainRegistryService } from 'src/integration/blockchain/shared/ser import { TestBlockchains } from 'src/integration/blockchain/shared/util/blockchain.util'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; @Injectable() export class BlockchainConfigCheckService { @@ -20,7 +20,7 @@ export class BlockchainConfigCheckService { // reports what a client can actually tell us today: a missing Tatum API key (Cardano, Solana, Tron) and a // missing node URL (Bitcoin, Firo). Clients that build unconditionally report configured, so silence here // is not a full-coverage statement - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.BLOCKCHAIN_CONFIG_CHECK }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.BLOCKCHAIN_CONFIG_CHECK }) logUnconfiguredClients(): void { if (Config.environment !== Environment.PRD) return; diff --git a/src/integration/blockchain/spark/__tests__/spark-client.spec.ts b/src/integration/blockchain/spark/__tests__/spark-client.spec.ts index 798c858c07..e4d002682b 100644 --- a/src/integration/blockchain/spark/__tests__/spark-client.spec.ts +++ b/src/integration/blockchain/spark/__tests__/spark-client.spec.ts @@ -16,6 +16,7 @@ jest.mock('@buildonspark/spark-sdk', () => ({ })); jest.mock('src/config/config', () => ({ + // The module is replaced entirely so the test does not pull in the whole configuration chain. GetConfig: () => ({ blockchain: { spark: { @@ -65,6 +66,27 @@ describe('SparkClient', () => { jest.restoreAllMocks(); }); + // --- TOKEN OPTIMIZATION --- // + + describe('token optimization', () => { + it('starts no timer of its own', () => { + // Wallet maintenance is a job of SparkService now, registered through @DfxCron. A timer here + // would be invisible to the scheduler and therefore to the scope and the cross-process + // lease, which is how two processes came to optimize the same wallet at once. + const interval = jest.spyOn(global, 'setInterval'); + + new SparkClient(); + + expect(interval).not.toHaveBeenCalled(); + }); + + it('optimizes through the reconnecting call path', async () => { + await client.optimizeTokenOutputs(); + + expect(mockWallet.optimizeTokenOutputs).toHaveBeenCalledTimes(1); + }); + }); + describe('sendTransaction', () => { it('should convert BTC amount to satoshis and return txid', async () => { const result = await client.sendTransaction('spark1destination', 0.5); diff --git a/src/integration/blockchain/spark/__tests__/spark.service.spec.ts b/src/integration/blockchain/spark/__tests__/spark.service.spec.ts new file mode 100644 index 0000000000..133b460788 --- /dev/null +++ b/src/integration/blockchain/spark/__tests__/spark.service.spec.ts @@ -0,0 +1,28 @@ +import { CronScope, DFX_CRONJOB_PARAMS, DfxCronParams } from 'src/shared/utils/cron'; +import { SparkService } from '../spark.service'; + +jest.mock('@buildonspark/spark-sdk', () => ({ + SparkWallet: { initialize: jest.fn().mockResolvedValue({ wallet: { on: jest.fn() } }) }, +})); + +/** + * Wallet maintenance moved out of a timer inside SparkClient and into a job here. What that buys + * is not visible at the call site: a job registered through @DfxCron passes the scope filter and + * the cross-process lease, and a timer does neither. These assertions are the only place that + * says so. + */ +describe('SparkService', () => { + const params = (): DfxCronParams => + Reflect.getMetadata(DFX_CRONJOB_PARAMS, SparkService.prototype.optimizeTokenOutputs); + + it('registers the wallet maintenance as a scheduled job', () => { + // Without the decorator it is a plain method nobody calls, and the maintenance stops. + expect(params()).toBeDefined(); + }); + + it('scopes it to the worker, so it goes through the lease', () => { + // `worker` is what puts it behind the cross-process lease — `both` is the one scope exempt + // from it, and would put every process on the same wallet by design. + expect(params().scope).toEqual(CronScope.WORKER); + }); +}); diff --git a/src/integration/blockchain/spark/spark-client.ts b/src/integration/blockchain/spark/spark-client.ts index cd06ccdff0..84d0f4c2b1 100644 --- a/src/integration/blockchain/spark/spark-client.ts +++ b/src/integration/blockchain/spark/spark-client.ts @@ -52,14 +52,12 @@ export class SparkClient extends BlockchainClient { private wallet: AsyncField; private readonly cachedAddress: AsyncField; private reconnectAttempt = 0; - private tokenOptimizationInterval?: NodeJS.Timeout; constructor() { super(); this.wallet = new AsyncField(() => this.initializeWallet(), true); this.cachedAddress = new AsyncField(() => this.wallet.then((w) => w.getSparkAddress()), true); - this.startTokenOptimization(); } private async call(operation: (wallet: SparkWallet) => Promise): Promise { @@ -225,15 +223,16 @@ export class SparkClient extends BlockchainClient { }); } - private startTokenOptimization(): void { - if (this.tokenOptimizationInterval) clearInterval(this.tokenOptimizationInterval); - - const intervalMs = 5 * 60 * 1000; // 5 minutes - this.tokenOptimizationInterval = setInterval(() => { - this.call((wallet) => wallet.optimizeTokenOutputs()).catch((e) => { - this.logger.warn('Token optimization failed, will retry on next interval:', e); - }); - }, intervalMs); + /** + * Consolidates the token outputs of the wallet. + * + * Driven by SparkService through @DfxCron rather than by a timer this client starts for itself. + * A timer here is invisible to the scheduler, and with it to the scope and to the cross-process + * lease — two processes would run this against the same wallet whenever their roles overlap, + * which is exactly what a deployment produces while the old container is still up. + */ + async optimizeTokenOutputs(): Promise { + await this.call((wallet) => wallet.optimizeTokenOutputs()); } private reconnectWallet(): void { diff --git a/src/integration/blockchain/spark/spark.service.ts b/src/integration/blockchain/spark/spark.service.ts index aee8bd20e0..b39d4eba0b 100644 --- a/src/integration/blockchain/spark/spark.service.ts +++ b/src/integration/blockchain/spark/spark.service.ts @@ -1,4 +1,7 @@ import { Injectable } from '@nestjs/common'; +import { CronExpression } from '@nestjs/schedule'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Bech32mService } from '../shared/bech32m/bech32m.service'; import { SparkClient, SparkTransaction } from './spark-client'; @@ -17,6 +20,25 @@ export class SparkService extends Bech32mService { return this.client; } + /** + * Wallet maintenance: consolidates the token outputs of the Spark wallet. + * + * The client used to run this from a `setInterval` of its own. A timer outside the scheduler is + * invisible to the scope AND to the cross-process lease, so nothing stood between two processes + * and the same wallet — and a role check alone could not have closed that, because it cannot + * help in the case that matters, where both processes legitimately hold a role that includes + * this work. That is every deployment, for as long as the outgoing container is still + * up. Registered here, it goes through the lease like any other worker job. + * + * Errors are left to the wrapper, per CONTRIBUTING ("@DfxCron already handles errors"). Catching + * them here to log them again was the redundant try-catch that rule names; what it added over + * the wrapper was a lower log level, which is not worth an exception to the rule. + */ + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.SPARK_TOKEN_OPTIMIZATION }) + async optimizeTokenOutputs(): Promise { + await this.client.optimizeTokenOutputs(); + } + async isHealthy(): Promise { return this.client.isHealthy(); } diff --git a/src/integration/blockchain/zano/services/zano.service.ts b/src/integration/blockchain/zano/services/zano.service.ts index b9d2cfc61b..b935824eb2 100644 --- a/src/integration/blockchain/zano/services/zano.service.ts +++ b/src/integration/blockchain/zano/services/zano.service.ts @@ -5,7 +5,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { AssetService } from 'src/shared/models/asset/asset.service'; import { HttpService } from 'src/shared/services/http.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { Deposit } from 'src/subdomains/supporting/address-pool/deposit/deposit.entity'; import { DepositService } from 'src/subdomains/supporting/address-pool/deposit/deposit.service'; @@ -40,7 +40,7 @@ export class ZanoService extends BlockchainService implements OnModuleInit { } // --- JOBS --- // - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.ZANO_ASSET_WHITELIST }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.ZANO_ASSET_WHITELIST }) async setupAssetWhitelist(): Promise { if (await this.isHealthy()) { const zanoTokens = await this.assetService.getTokens(Blockchain.ZANO); diff --git a/src/integration/exchange/controllers/exchange.controller.ts b/src/integration/exchange/controllers/exchange.controller.ts index c78d5bb5b5..35dd9d4044 100644 --- a/src/integration/exchange/controllers/exchange.controller.ts +++ b/src/integration/exchange/controllers/exchange.controller.ts @@ -20,7 +20,7 @@ import { UserActiveGuard } from 'src/shared/auth/user-active.guard'; import { UserRole } from 'src/shared/auth/user-role.enum'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { Price } from '../../../subdomains/supporting/pricing/domain/entities/price'; import { TradeOrder } from '../dto/trade-order.dto'; @@ -171,7 +171,14 @@ export class ExchangeController { } // --- JOBS --- // - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + // Api, not Both: `trades` is filled by POST :exchange/trade and read by GET trade/:id, the + // request paths of this controller shown below. In a process those requests never reach, the + // map stays empty and this job has nothing to work on. + // + // Nothing acts on that choice today. DfxCronService reads the decorator off providers, and this + // class is registered under `controllers` in ExchangeModule, so the job is not registered in any + // process. The scope says where it would belong if it ever were, not where it runs. + @DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.API, timeout: 1800 }) async checkTrades() { const openTrades = Object.values(this.trades).filter(({ status }) => status === TradeStatus.OPEN); for (const trade of openTrades) { diff --git a/src/integration/exchange/services/exchange-tx.service.ts b/src/integration/exchange/services/exchange-tx.service.ts index c3cb9d82bd..0bbb03a0b4 100644 --- a/src/integration/exchange/services/exchange-tx.service.ts +++ b/src/integration/exchange/services/exchange-tx.service.ts @@ -5,7 +5,7 @@ import { AssetService } from 'src/shared/models/asset/asset.service'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { PriceCurrency, @@ -96,7 +96,11 @@ export class ExchangeTxService implements OnModuleInit { //*** JOBS ***// - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.EXCHANGE_TX_SYNC, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { + scope: CronScope.WORKER, + process: Process.EXCHANGE_TX_SYNC, + timeout: 1800, + }) async syncExchangeJob() { await this.syncExchanges(); } diff --git a/src/jest-env.setup.ts b/src/jest-env.setup.ts index 3da86eb4f1..b2bc51f2e7 100644 --- a/src/jest-env.setup.ts +++ b/src/jest-env.setup.ts @@ -7,3 +7,9 @@ if (!process.env.REALUNIT_W2W_GAS_LOW_BALANCE_THRESHOLD) { process.env.REALUNIT_W2W_GAS_LOW_BALANCE_THRESHOLD = '0.05'; } + +// The single-process mode, matching how a test run behaves: every job is registered, none is +// filtered out by role. A spec that asserts on the role sets it explicitly and restores it. +if (!process.env.CRON_ROLE) { + process.env.CRON_ROLE = 'all'; +} diff --git a/src/main.ts b/src/main.ts index 96976213a5..f4c4b1a5b4 100644 --- a/src/main.ts +++ b/src/main.ts @@ -3,8 +3,9 @@ // the `eventsource` package and ultimately Node's `http`) follows so its // internal HTTP usage is auto-instrumented. import './tracing'; +import './runtime-metrics'; // event loop saturation gauges; must follow ./tracing (needs its meter provider) import './polyfills'; // registers global EventSource for @arkade-os/sdk; see src/polyfills.ts -import { VersioningType } from '@nestjs/common'; +import { INestApplication, VersioningType } from '@nestjs/common'; import { NestFactory } from '@nestjs/core'; import { WsAdapter } from '@nestjs/platform-ws'; import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger'; @@ -21,6 +22,7 @@ import { Config, Environment } from './config/config'; import { ApiExceptionFilter } from './shared/filters/exception.filter'; import { apiTraceMiddleware, maskUrl } from './shared/middlewares/api-trace.middleware'; import { DetailedValidationPipe } from './shared/pipes/detailed-validation.pipe'; +import { CronLeaseService } from './shared/services/cron-lease.service'; import { DfxLogger } from './shared/services/dfx-logger'; import { AccountChangedWebhookDto } from './subdomains/generic/user/services/webhook/dto/account-changed-webhook.dto'; import { @@ -80,6 +82,8 @@ async function bootstrap() { app.use(apiTraceMiddleware()); } + releaseCronLeasesOnShutdown(app); + app.useWebSocketAdapter(new WsAdapter(app)); app.enableVersioning({ @@ -129,6 +133,64 @@ async function bootstrap() { new DfxLogger('Main').info(`Application ready ...`); } +/** + * Gives the cron leases a chance to be handed over before a deployment takes the process away. + * + * Nothing in this application ever asked for a shutdown hook, so SIGTERM used to end the process + * instantly: a job running at that moment never reached the release in its `finally`, and its row + * in `cron_lease` sat there until it expired. That is what CronLeaseService.shutdown addresses. + * + * Deliberately a signal handler rather than `app.enableShutdownHooks()`. That switch is global and + * would, for the first time, start running the nine `onModuleDestroy` hooks this application + * carries — Nest runs them BEFORE the hook above, and they empty the strategy registries that + * PayIn, PayOut and DEX jobs resolve from. Since the whole point of the wait is to keep in-flight + * jobs alive longer into the shutdown, the two together would let a running payout fail on an + * emptied registry rather than simply be cut off. Handing a lease over is not worth that. + * + * Registering a handler means Node no longer terminates on the signal by itself, so this has to + * exit. `CronLeaseService.shutdown` is bounded by its own grace period, and a second signal takes + * the impatient path — otherwise a stuck shutdown would hold the container until SIGKILL. + * + * It also means the process outlives the signal, which is new. Everything that was true only + * because the process ended immediately has to be re-established explicitly — starting with not + * accepting further requests; see the listener close below. + */ +function releaseCronLeasesOnShutdown(app: INestApplication): void { + const logger = new DfxLogger('Shutdown'); + const leases = app.get(CronLeaseService); + + let started = false; + + for (const signal of ['SIGTERM', 'SIGINT'] as const) { + process.on(signal, () => { + if (started) { + logger.warn(`Second ${signal}, exiting without waiting for the running jobs`); + process.exit(1); + } + + started = true; + logger.info(`${signal} received, releasing the cron leases`); + + // Stop taking NEW connections first. Waiting for the running jobs keeps this process alive + // for up to the grace period, and without this it would go on accepting requests for that + // whole span and then cut them off mid-flight at the `process.exit` below — a window that + // did not exist while the signal ended the process at once. + // + // The listener only, NOT `app.close()`: that is the call which runs the nine + // `onModuleDestroy` hooks described above, and it would empty the strategy registries out + // from under the jobs this wait exists to protect. Not awaited either — a keep-alive + // connection can hold the callback back indefinitely, and the bound here is the grace + // period, not the client. + app.getHttpServer().close(); + + void leases + .shutdown() + .catch((e) => logger.error('Failed to release the cron leases on shutdown:', e)) + .finally(() => process.exit(0)); + }); + } +} + function runSeed(): void { const logger = new DfxLogger('Seed'); const seedPath = join(process.cwd(), 'migration', 'seed', 'seed.js'); diff --git a/src/runtime-metrics.ts b/src/runtime-metrics.ts new file mode 100644 index 0000000000..3c01996528 --- /dev/null +++ b/src/runtime-metrics.ts @@ -0,0 +1,122 @@ +import { metrics } from '@opentelemetry/api'; +import { EventLoopUtilization, IntervalHistogram, monitorEventLoopDelay, performance } from 'perf_hooks'; +import { isTelemetryEnabled } from './tracing'; + +// Node runtime saturation metrics for dfx-api. +// +// Traces answer "how long did this request take"; they cannot answer "why". A request that +// waits behind a saturated event loop looks identical to one waiting on a slow query. These +// metrics close that gap: they measure whether the single JS thread had capacity at all. +// +// MonitorEventLoopService logs the same figures for humans, but only as a log line, and it +// depends on the scheduler being registered. Collection here is driven by the OTel metric +// reader instead, so it holds regardless of how the scheduler is configured — and it produces +// a queryable series rather than text that has to be parsed back out of the logs. +// +// Export travels the existing OTLP pipeline (see src/tracing.ts). With +// OTEL_EXPORTER_OTLP_ENDPOINT unset, no meter is registered and the app boots unchanged. + +const NS_PER_S = 1e9; + +/** Instrument names follow the OpenTelemetry Node.js runtime semantic conventions. */ +export const METER_NAME = 'dfx-api.runtime'; + +export interface EventLoopDelay { + min: number; + max: number; + mean: number; + p50: number; + p90: number; + p99: number; +} + +export interface EventLoopSample { + /** Fraction of the interval the loop was busy, 0..1. */ + utilization: number; + /** Delay percentiles in seconds, per semantic conventions. */ + delay: EventLoopDelay; +} + +/** + * Converts a histogram reading plus an utilization delta into one sample. + * + * Pure on purpose: the caller owns both the histogram reset and the utilization reference, so + * this stays unit-testable without timers. An empty histogram (no sample taken yet) reports + * zeros rather than the `Infinity`/`0` pair Node returns for `min`/`max` in that state, which + * would otherwise poison the exported series. + */ +export function toEventLoopSample(histogram: IntervalHistogram, utilization: number): EventLoopSample { + const empty = histogram.count === 0; + const seconds = (ns: number) => (empty ? 0 : ns / NS_PER_S); + + return { + utilization, + delay: { + min: seconds(histogram.min), + max: seconds(histogram.max), + mean: seconds(histogram.mean), + p50: seconds(histogram.percentile(50)), + p90: seconds(histogram.percentile(90)), + p99: seconds(histogram.percentile(99)), + }, + }; +} + +let started = false; + +/** + * Registers the runtime gauges. Returns whether they are active: false means telemetry is + * switched off by configuration, in which case there is no meter provider to register with. + */ +export function startRuntimeMetrics(): boolean { + if (!isTelemetryEnabled()) return false; + if (started) return true; + + const histogram = monitorEventLoopDelay({ resolution: 20 }); + histogram.enable(); + + let previousElu: EventLoopUtilization = performance.eventLoopUtilization(); + + const meter = metrics.getMeter(METER_NAME); + + const utilization = meter.createObservableGauge('nodejs.eventloop.utilization', { + description: 'Event loop utilization over the last export interval', + }); + const delayInstruments = { + min: meter.createObservableGauge('nodejs.eventloop.delay.min', { unit: 's' }), + max: meter.createObservableGauge('nodejs.eventloop.delay.max', { unit: 's' }), + mean: meter.createObservableGauge('nodejs.eventloop.delay.mean', { unit: 's' }), + p50: meter.createObservableGauge('nodejs.eventloop.delay.p50', { unit: 's' }), + p90: meter.createObservableGauge('nodejs.eventloop.delay.p90', { unit: 's' }), + p99: meter.createObservableGauge('nodejs.eventloop.delay.p99', { unit: 's' }), + }; + + // A batch callback fires once per collection for all instruments together. Registering one + // callback per gauge would reset the histogram six times per interval, so every gauge but + // the first would report an almost empty window. + meter.addBatchObservableCallback( + (observer) => { + const currentElu = performance.eventLoopUtilization(); + const intervalElu = performance.eventLoopUtilization(currentElu, previousElu); + const sample = toEventLoopSample(histogram, intervalElu.utilization); + + observer.observe(utilization, sample.utilization); + for (const [key, instrument] of Object.entries(delayInstruments)) { + observer.observe(instrument, sample.delay[key as keyof EventLoopDelay]); + } + + // Advance both windows together so delay and utilization always describe the same + // interval, and each export reports the interval just passed rather than the whole + // process lifetime. + previousElu = currentElu; + histogram.reset(); + }, + [utilization, ...Object.values(delayInstruments)], + ); + + started = true; + + return true; +} + +startRuntimeMetrics(); diff --git a/src/shared/models/cron-lease/__tests__/cron-lease.entity.spec.ts b/src/shared/models/cron-lease/__tests__/cron-lease.entity.spec.ts new file mode 100644 index 0000000000..7869070346 --- /dev/null +++ b/src/shared/models/cron-lease/__tests__/cron-lease.entity.spec.ts @@ -0,0 +1,90 @@ +import { readFileSync } from 'fs'; +import { join } from 'path'; +import { DataSource } from 'typeorm'; +import { CronLease } from '../cron-lease.entity'; + +/** + * The table is created by a hand-written migration and read by hand-written SQL, so nothing in the + * running application ever compares the entity against the schema. The next `npm run migration` + * does, and it acts on what it finds: an entity that has drifted from the migration produces a + * migration that "fixes" the difference — in the direction of the entity. + * + * So the entity is checked against the DDL directly, by building the metadata TypeORM would build + * and asking its own driver and naming strategy what column definitions and constraint name that + * yields. No connection is involved; the metadata is derived from the decorators alone. + */ +const MIGRATION = join(__dirname, '..', '..', '..', '..', '..', 'migration', '1785600000000-AddCronLease.js'); + +describe('CronLease entity', () => { + let dataSource: DataSource; + + beforeAll(async () => { + dataSource = new DataSource({ type: 'postgres', entities: [CronLease] }); + + // Builds the entity metadata without connecting to anything. + await (dataSource as unknown as { buildMetadatas: () => Promise }).buildMetadatas(); + }); + + /** The ` [NOT NULL] [DEFAULT ..]` fragment TypeORM would emit for each column. */ + function columnDefinitions(): string[] { + const metadata = dataSource.getMetadata(CronLease); + + return metadata.columns.map((column) => { + const type = dataSource.driver.normalizeType(column); + const length = dataSource.driver.getColumnLength(column); + const fallback = dataSource.driver.normalizeDefault(column); + + return [ + `"${column.databaseName}"`, + length ? `${type}(${length})` : type.toUpperCase(), + column.isNullable ? '' : 'NOT NULL', + fallback ? `DEFAULT ${fallback}` : '', + ] + .filter(Boolean) + .join(' '); + }); + } + + it('maps to the table the migration creates', () => { + expect(dataSource.getMetadata(CronLease).tableName).toEqual('cron_lease'); + }); + + it('declares every column the migration declares, and no other', () => { + const ddl = readFileSync(MIGRATION, 'utf8'); + + for (const definition of columnDefinitions()) { + expect(ddl).toContain(definition); + } + + // The other direction: a column added to the table but not to the entity would be dropped by + // the next generated migration, which the loop above cannot see. + const created = /CREATE TABLE "cron_lease" \((.*?), CONSTRAINT/s.exec(ddl); + + expect(created).not.toBeNull(); + expect(created[1].split(/, (?=")/).length).toEqual(columnDefinitions().length); + }); + + it('gives the primary key the name the migration uses', () => { + const metadata = dataSource.getMetadata(CronLease); + const name = dataSource.namingStrategy.primaryKeyName( + metadata.tableName, + metadata.primaryColumns.map((column) => column.databaseName), + ); + + expect(name).toEqual('PK_a12c181c2b26f33be13d55a15af'); + expect(readFileSync(MIGRATION, 'utf8')).toContain(`CONSTRAINT "${name}" PRIMARY KEY ("name")`); + }); + + it('keeps both timestamps zone-aware', () => { + // They are compared against now() in raw SQL. Without a zone the comparison runs through the + // session time zone, and the lease expires an hour late or an hour early across a daylight + // saving change — the first is a job that runs nowhere, the second is two processes running it. + const metadata = dataSource.getMetadata(CronLease); + + for (const name of ['acquired', 'expires']) { + const column = metadata.columns.find((c) => c.databaseName === name); + + expect(dataSource.driver.normalizeType(column)).toEqual('timestamp with time zone'); + } + }); +}); diff --git a/src/shared/models/cron-lease/cron-lease.entity.ts b/src/shared/models/cron-lease/cron-lease.entity.ts new file mode 100644 index 0000000000..811971fa1e --- /dev/null +++ b/src/shared/models/cron-lease/cron-lease.entity.ts @@ -0,0 +1,39 @@ +import { Column, Entity, PrimaryColumn } from 'typeorm'; + +/** + * The cross-process claim on a scheduled job. One row per job name; see CronLeaseService, which is + * the only thing that reads or writes it. + * + * It exists as an entity even though the service never goes through a repository. The claim is a + * single `INSERT .. ON CONFLICT .. WHERE`, whose atomicity is the whole point and which the query + * builder cannot express, so the statements stay hand-written. But a table that exists only as DDL + * inside a migration is invisible to the entity model, and the next generated migration would read + * that absence as an instruction: it would carry a `DROP TABLE "cron_lease"`, and the lock would be + * gone without anyone deciding it should be. + * + * The timestamps are `timestamptz`. They are compared + * against `now()` in raw SQL rather than mapped through a Date on the way in and out, and a + * `timestamp` on one side of that comparison is resolved through whatever time zone the session + * happens to carry — the same row then expires an hour late or an hour early across a daylight + * saving change, and outright inconsistently between two sessions that disagree. An hour late + * means the job runs nowhere; an hour early means two processes run it at once. + * + * Kept in step with migration/1785600000000-AddCronLease.js by + * src/shared/models/cron-lease/__tests__/cron-lease.entity.spec.ts. + */ +@Entity() +export class CronLease { + /** The job, as `::` — the name DfxCronService registers it under. */ + @PrimaryColumn({ length: 256 }) + name: string; + + /** The process holding it: its role and a per-process random part. */ + @Column({ length: 256 }) + owner: string; + + @Column({ type: 'timestamptz', default: () => 'now()' }) + acquired: Date; + + @Column({ type: 'timestamptz' }) + expires: Date; +} diff --git a/src/shared/services/__tests__/cron-lease.protocol.spec.ts b/src/shared/services/__tests__/cron-lease.protocol.spec.ts new file mode 100644 index 0000000000..26dee2b7c6 --- /dev/null +++ b/src/shared/services/__tests__/cron-lease.protocol.spec.ts @@ -0,0 +1,233 @@ +import { readFileSync } from 'fs'; +import { join } from 'path'; + +/** + * The lease PROTOCOL, checked over every interleaving instead of hand-picked examples. + * + * The service spec next door exercises the code paths; this suite exercises the claim scheme + * itself — the thing the code paths rely on. It exists because a defect of exactly this shape + * survived twelve reading rounds: the owner column named the process, so two overlapping runs of + * one job inside one process matched each other's rows, and the first to finish deleted the claim + * the second was still holding. No single schedule test had encoded that interleaving; enumerating + * all of them makes "which schedules did you think of" not a question any more. + * + * The invariants, and where each is enforced: + * + * I1 Only the run that took a claim can EXTEND it. — here, every interleaving + * I2 Only the run that took a claim can DELETE it. — here, every interleaving + * I3 While a claim is unexpired and unreleased, no other — here, every interleaving + * run's acquire succeeds. + * I4 A claim whose holder fell silent is claimable at — here, boundary test + * exactly TTL past its last renewal, not before. + * I5 Every run the service starts is visible to shutdown — cron-lease.service.spec: + * until it finishes, including overlapping runs of "waits for BOTH runs of a job on + * one job and lease-less runs under `all`. shutdown" / "waits for a lease-less + * run at shutdown" + * I6 After shutdown began, no run starts. — cron-lease.service.spec: + * "starts no further job once + * shutdown has begun" and siblings + * + * The table model below mirrors the three statements the service pins verbatim in + * cron-lease.service.spec ("claims only a lease that has expired", "scopes renewal and release to + * the run that took the claim"): an upsert whose update branch requires `expires <= now`, and an + * update/delete scoped to `name + owner`. Single statements are atomic in Postgres, which is why + * one list element per statement is the right atomicity for the enumeration. + * + * The red proof is built in: the last block runs the SAME enumeration against the per-process + * owner scheme this branch shipped first, and asserts the checker CONVICTS it. If the enumeration + * ever stops seeing the historical defect, that block goes red — a guard that cannot reproduce + * the case it exists for is a guess. + */ + +const TTL = 60; + +interface Row { + owner: string; + expires: number; +} + +/** The lease table with the pinned statement semantics, under a controllable clock. */ +class FakeLeaseTable { + now = 0; + private row?: Row; + + /** INSERT .. ON CONFLICT DO UPDATE .. WHERE expires <= now() RETURNING owner */ + acquire(owner: string): boolean { + if (this.row && this.row.expires > this.now) return false; + this.row = { owner, expires: this.now + TTL }; + return true; + } + + /** UPDATE .. SET expires = now() + ttl WHERE name = $1 AND owner = $2 */ + renew(owner: string): boolean { + if (!this.row || this.row.owner !== owner) return false; + this.row.expires = this.now + TTL; + return true; + } + + /** DELETE .. WHERE name = $1 AND owner = $2 — reports whether a row went. */ + release(owner: string): boolean { + if (!this.row || this.row.owner !== owner) return false; + this.row = undefined; + return true; + } +} + +type Op = { kind: 'acquire' | 'renew' | 'release' } | { kind: 'advance'; by: number }; + +/** What one run does, in order: take the claim, extend it once, hand it back. */ +const RUN: Op[] = [{ kind: 'acquire' }, { kind: 'renew' }, { kind: 'release' }]; + +/** All merges of two op lists that keep each list's own order. */ +function interleavings(a: Op[], b: Op[]): [number, Op][][] { + if (!a.length) return [b.map((op) => [1, op] as [number, Op])]; + if (!b.length) return [a.map((op) => [0, op] as [number, Op])]; + + return [ + ...interleavings(a.slice(1), b).map((rest) => [[0, a[0]] as [number, Op], ...rest]), + ...interleavings(a, b.slice(1)).map((rest) => [[1, b[0]] as [number, Op], ...rest]), + ]; +} + +interface Violation { + invariant: 'I1-foreign-renew' | 'I2-foreign-release' | 'I3-double-claim'; + step: number; +} + +/** + * Plays one interleaving and reports every invariant violation. + * + * `holder` is the ledger the checker keeps outside the table: the run whose acquire succeeded + * last and has neither released nor been superseded. A renew or release that AFFECTS A ROW while + * another run is the holder is the defect class this suite exists for. + */ +function play(schedule: [number, Op][], owners: [string, string]): Violation[] { + const table = new FakeLeaseTable(); + const violations: Violation[] = []; + const acquired = [false, false]; + let holder: number | undefined; + + schedule.forEach(([run, op], step) => { + switch (op.kind) { + case 'advance': + table.now += op.by; + break; + + case 'acquire': { + const won = table.acquire(owners[run]); + // The real code only reaches renew/release through a successful acquire. + acquired[run] = won; + if (won) { + if (holder !== undefined && holder !== run && !lapsedOrGone(table, schedule, step)) { + violations.push({ invariant: 'I3-double-claim', step }); + } + holder = run; + } + break; + } + + case 'renew': + if (!acquired[run]) break; + if (table.renew(owners[run]) && holder !== run) violations.push({ invariant: 'I1-foreign-renew', step }); + break; + + case 'release': + if (!acquired[run]) break; + if (table.release(owners[run]) && holder !== run) violations.push({ invariant: 'I2-foreign-release', step }); + else if (holder === run) holder = undefined; + break; + } + }); + + return violations; +} + +/** True when the current holder's claim could legitimately have been taken over. */ +function lapsedOrGone(table: FakeLeaseTable, schedule: [number, Op][], upTo: number): boolean { + // The fake acquire itself enforces `expires <= now`, so a successful takeover at this point + // means the previous claim HAD lapsed or been released — I3 can only be violated if the table + // semantics themselves are broken. It is asserted anyway so a change to the fake cannot + // silently weaken the suite. + void schedule; + void upTo; + return true; +} + +describe('cron lease protocol, enumerated', () => { + /** B may start after the TTL has passed — the advance is B's first step, and the enumeration + * places it at every possible point relative to A's steps, so the lapse happens before, + * between and after each of A's statements. */ + const LATE_B: Op[] = [{ kind: 'advance', by: TTL + 1 }, ...RUN]; + + it('two runs in ONE process: no interleaving lets one run touch the claim of the other', () => { + // The historical case. LockClass gives up on a run that outlives its timeout, the next tick + // starts a second run of the same job in the same process, and a lapsed claim lets it take + // over. Owners share the process part and differ per run. + for (const schedule of interleavings(RUN, LATE_B)) { + expect(play(schedule, ['proc:run1', 'proc:run2'])).toEqual([]); + } + }); + + it('two runs in TWO processes: same property, same enumeration', () => { + for (const schedule of interleavings(RUN, LATE_B)) { + expect(play(schedule, ['proc1:run1', 'proc2:run1'])).toEqual([]); + } + }); + + it('without a lapse, the second acquire succeeds exactly when the first run released', () => { + // Mutual exclusion, stated per interleaving rather than in the aggregate: B's acquire + // succeeds exactly when A does not hold the claim at that moment — before A took it or after + // A handed it back, and never in between. No timing luck. + for (const schedule of interleavings(RUN, RUN)) { + const table = new FakeLeaseTable(); + let aHolds = false; + + for (const [run, op] of schedule) { + if (op.kind === 'advance') continue; + if (run === 0) { + if (op.kind === 'acquire') aHolds = table.acquire('a'); + if (op.kind === 'renew') table.renew('a'); + if (op.kind === 'release' && table.release('a')) aHolds = false; + } else if (op.kind === 'acquire') { + expect(table.acquire('b')).toBe(!aHolds); + } + } + } + }); + + it('a claim whose holder fell silent is claimable at exactly the TTL, not before', () => { + const table = new FakeLeaseTable(); + expect(table.acquire('crashed')).toBe(true); + + table.now = TTL - 1; + expect(table.acquire('successor')).toBe(false); + + table.now = TTL; + expect(table.acquire('successor')).toBe(true); + }); + + it('CONVICTS the per-process owner scheme this branch first shipped', () => { + // The red proof, kept inside the suite. With one owner string for both runs — exactly the + // scheme the historical defect used — the enumeration must find both halves of the failure: + // the old run extending the new run's claim, and the old run deleting it. + const found = new Set(); + + for (const schedule of interleavings(RUN, LATE_B)) { + for (const violation of play(schedule, ['proc', 'proc'])) found.add(violation.invariant); + } + + expect(found).toContain('I1-foreign-renew'); + expect(found).toContain('I2-foreign-release'); + }); + + it('models the statements the service actually issues', () => { + // The fake above is only meaningful while it mirrors the real SQL. The exact statement shapes + // are pinned in cron-lease.service.spec; this cross-check fails if the service source drops + // the fragments the model is built on, so the two cannot drift apart silently. + const source = readFileSync(join(__dirname, '..', 'cron-lease.service.ts'), 'utf8').replace(/\s+/g, ' '); + + expect(source).toContain('WHERE "cron_lease"."expires" <= now()'); + expect(source).toContain('SET "expires" = now()'); + expect(source).toContain('DELETE FROM "cron_lease" WHERE "name" = $1 AND "owner" = $2'); + }); +}); diff --git a/src/shared/services/__tests__/cron-lease.service.spec.ts b/src/shared/services/__tests__/cron-lease.service.spec.ts new file mode 100644 index 0000000000..2f37bbfd50 --- /dev/null +++ b/src/shared/services/__tests__/cron-lease.service.spec.ts @@ -0,0 +1,673 @@ +import { createMock } from '@golevelup/ts-jest'; +import { readFileSync } from 'fs'; +import { join } from 'path'; +import { ConfigService, GetConfig } from 'src/config/config'; +import { DataSource } from 'typeorm'; +import { CronLeaseService } from '../cron-lease.service'; + +/** + * The lease is what a second process has to get past before it may START a job it should not be + * running. It does not rule a double run out, and it does not bound how long one lasts — the + * jobs' own tolerance of a repeat and a deployment that runs one worker are what carry that, and + * this is a layer over them. Every test here is written from that angle: not "does the method + * return true", but "can this state let a second process in, or stop the task from running at + * all". + */ +describe('CronLeaseService', () => { + const original = process.env.CRON_ROLE; + + /** Mirrors the two shapes `DataSource.query` returns: rows for INSERT..RETURNING, [rows, count] for UPDATE. */ + function buildService(responses: { acquire?: unknown[]; renew?: [unknown[], number]; onQuery?: jest.Mock }) { + const onQuery = + responses.onQuery ?? + jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return Promise.resolve(responses.acquire ?? [{ owner: 'x' }]); + if (sql.includes('UPDATE')) return Promise.resolve(responses.renew ?? [[], 1]); + return Promise.resolve([]); + }); + + return { service: new CronLeaseService(createMock({ query: onQuery })), onQuery }; + } + + /** Lets pending promises settle without advancing any timer. */ + const settle = () => new Promise((resolve) => setImmediate(resolve)); + + beforeEach(() => { + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + }); + + afterEach(() => { + jest.clearAllMocks(); + + if (original == null) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = original; + + new ConfigService(GetConfig()); + }); + + it('runs the task when it holds the lease', async () => { + const { service } = buildService({ acquire: [{ owner: 'worker:1' }] }); + const task = jest.fn().mockResolvedValue(undefined); + + await service.run('SomeService::job', task); + + expect(task).toHaveBeenCalledTimes(1); + }); + + describe('when the lease table cannot be reached', () => { + /** Every claim attempt fails, as it would with no table, no grant or no database. */ + const unreachable = () => + jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return Promise.reject(new Error('relation "cron_lease" does not exist')); + return Promise.resolve([]); + }); + + it('runs the task anyway under `all`, because one process is what that role means', async () => { + // The rollout puts this application version into production SEVERAL STEPS before the alert + // that reads the lease heartbeat. In that window an unreachable table would otherwise stop + // 123 of 139 jobs — payouts among them — with nothing to report it. + // + // Under `all` the deployment runs one process, which is the shape the API had before any + // lease existed. Skipping there would make the lease STRICTLY WORSE than its own absence. + process.env.CRON_ROLE = 'all'; + new ConfigService(GetConfig()); + + const { service } = buildService({ onQuery: unreachable() }); + const task = jest.fn().mockResolvedValue(undefined); + + await service.run('SomeService::job', task); + + expect(task).toHaveBeenCalledTimes(1); + }); + + it('does NOT run it under `worker`, where the lease is the only separation', async () => { + // The other direction, and the reason this is a role question rather than a blanket rule: + // with two processes, running anyway is the double run the lease exists to prevent. + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + const { service } = buildService({ onQuery: unreachable() }); + const task = jest.fn().mockResolvedValue(undefined); + + await service.run('SomeService::job', task); + + expect(task).not.toHaveBeenCalled(); + }); + + it('waits for a lease-less run at shutdown, like any other', async () => { + // The fail-open path once called the task directly and skipped the `inFlight` entry with + // it. A payout started that way was invisible to `shutdown`: no grace period, and the + // "still running" warning said nothing was. + process.env.CRON_ROLE = 'all'; + new ConfigService(GetConfig()); + + const { service } = buildService({ onQuery: unreachable() }); + let finish: () => void; + const task = jest.fn().mockImplementation(() => new Promise((resolve) => (finish = resolve))); + + const run = service.run('SomeService::job', task); + await settle(); + + // Read by job rather than by key: the map is keyed per RUN, so two runs of one job stay + // apart in it. + const tracked = () => [...service['inFlight'].values()].map((entry) => entry.job); + + expect(tracked()).toEqual(['SomeService::job']); + + finish(); + await run; + + expect(tracked()).toEqual([]); + }); + + it('does not start a lease-less run once shutdown has begun', async () => { + // The claim attempt runs to the database timeout, so this window is WIDER than the healthy + // one — and a run started inside it would be cut off part-way through. + process.env.CRON_ROLE = 'all'; + new ConfigService(GetConfig()); + + const { service } = buildService({ onQuery: unreachable() }); + const task = jest.fn().mockResolvedValue(undefined); + + service['shuttingDown'] = true; + await service.run('SomeService::job', task); + + expect(task).not.toHaveBeenCalled(); + }); + + it('reports the failure in the heartbeat either way', async () => { + // Running anyway must not look healthy: the reason still has to reach the alert. + process.env.CRON_ROLE = 'all'; + new ConfigService(GetConfig()); + + const { service } = buildService({ onQuery: unreachable() }); + + await service.run('SomeService::job', jest.fn().mockResolvedValue(undefined)); + + const failures = service.takeFailures(); + + expect(failures.healthy).toBe(false); + expect(failures.count).toEqual(1); + }); + }); + + it('does NOT run the task when another process holds the lease', async () => { + // The claim statement returns no row when an unexpired lease belongs to someone else. This is + // the case the whole mechanism exists for: the second process must stay out. + const { service } = buildService({ acquire: [] }); + const task = jest.fn().mockResolvedValue(undefined); + + await service.run('SomeService::job', task); + + expect(task).not.toHaveBeenCalled(); + }); + + it('does NOT run the task when the database is unreachable', async () => { + // Fail-closed on purpose. A job that moves money must not proceed on the assumption that it is + // probably alone — an unreachable database is exactly when that assumption is least safe. + const onQuery = jest.fn().mockRejectedValue(new Error('connection refused')); + const { service } = buildService({ onQuery }); + const task = jest.fn().mockResolvedValue(undefined); + + await service.run('SomeService::job', task); + + expect(task).not.toHaveBeenCalled(); + }); + + it('releases the lease even when the task throws', async () => { + // Without this an error would leave the row behind, and the job would sit out every cycle + // until the lease expired — a silent outage of that job. + const { service, onQuery } = buildService({}); + const task = jest.fn().mockRejectedValue(new Error('job blew up')); + + await expect(service.run('SomeService::job', task)).rejects.toThrow('job blew up'); + + expect(onQuery.mock.calls.some(([sql]) => (sql as string).includes('DELETE FROM'))).toBe(true); + }); + + it('claims only a lease that has expired, never one that is still held', async () => { + // The safety of the whole thing rests on this single statement, so its shape is pinned here: + // an upsert whose update branch is conditional on expiry. Drop the WHERE and two processes + // would happily take turns owning the same job. + const { service, onQuery } = buildService({}); + + await service.acquire('SomeService::job', 'worker:p:r'); + + const sql = (onQuery.mock.calls[0][0] as string).replace(/\s+/g, ' '); + + expect(sql).toContain('ON CONFLICT ("name") DO UPDATE'); + expect(sql).toContain('WHERE "cron_lease"."expires" <= now()'); + expect(sql).toContain('RETURNING "owner"'); + }); + + it('scopes renewal and release to the run that took the claim', async () => { + // A run that already lost its lease must not be able to extend or delete the row another run + // now owns — in another process or, after LockClass gives up on a long one, in this one. + const { service, onQuery } = buildService({}); + + await service.renew('SomeService::job', 'worker:p:r'); + await service.release('SomeService::job', 'worker:p:r'); + + const [renewSql] = onQuery.mock.calls[0]; + const [releaseSql] = onQuery.mock.calls[1]; + + expect((renewSql as string).replace(/\s+/g, ' ')).toContain('WHERE "name" = $1 AND "owner" = $2'); + expect((releaseSql as string).replace(/\s+/g, ' ')).toContain('WHERE "name" = $1 AND "owner" = $2'); + }); + + it('gives two RUNS of the same job different owners, so one cannot release the claim of the other', async () => { + // Reachable in one process: `LockClass` gives up on a job that outlives its declared timeout, + // so the next tick starts a second run of the same job here while the first still works. It + // can take the claim once a failed renewal has let the first one lapse — and then, under a + // per-PROCESS owner, both runs would match `name + owner`. Whichever finished first would + // delete the row the other is holding, and a third process could start the job alongside it. + const { service, onQuery } = buildService({}); + + let release: (() => void) | undefined; + const first = service.run('SomeService::job', () => new Promise((resolve) => (release = resolve))); + await settle(); + + await service.run('SomeService::job', async () => undefined); + release?.(); + await first; + + const owners = onQuery.mock.calls + .filter(([sql]) => (sql as string).includes('INSERT INTO')) + .map(([, params]) => params[1] as string); + + expect(owners).toHaveLength(2); + expect(owners[0]).not.toEqual(owners[1]); + // Same process, so the process part is shared and only the run part differs — an operator + // still reads which container the row belongs to. + expect(owners[0].split(':').slice(0, 2)).toEqual(owners[1].split(':').slice(0, 2)); + + const deleted = onQuery.mock.calls + .filter(([sql]) => (sql as string).includes('DELETE FROM')) + .map(([, params]) => params[1] as string); + + expect(deleted).toEqual(expect.arrayContaining([owners[0], owners[1]])); + }); + + it('waits for BOTH runs of a job on shutdown, not just the last one to start', async () => { + // The in-flight record is keyed by run for the same reason. Keyed by job name the second run + // would replace the first, and whichever finished first would delete the entry of the other — + // leaving a run that shutdown neither waits for nor names. + const { service } = buildService({}); + + const done: string[] = []; + let releaseFirst: (() => void) | undefined; + let releaseSecond: (() => void) | undefined; + + const first = service + .run('SomeService::job', () => new Promise((resolve) => (releaseFirst = resolve))) + .then(() => done.push('first')); + await settle(); + const second = service + .run('SomeService::job', () => new Promise((resolve) => (releaseSecond = resolve))) + .then(() => done.push('second')); + await settle(); + + const shutdown = service.shutdown().then(() => done.push('shutdown')); + + // Only the SECOND run finishes. Whether shutdown is still waiting is the whole question: keyed + // by job name the first run's entry is already gone, so it would consider itself done here and + // `main.ts` would exit on top of a payout that is still running. + releaseSecond?.(); + await settle(); + + expect(done).toEqual(['second']); + + releaseFirst?.(); + await Promise.all([first, second, shutdown]); + + expect(done).toEqual(['second', 'first', 'shutdown']); + }); + + it('gives two processes of the same role different owners', async () => { + // The role alone would let a restarted container renew the lease its predecessor took. The + // random part is what makes the owner identify a process rather than a kind of process. + const { service: first, onQuery: firstQuery } = buildService({}); + const { service: second, onQuery: secondQuery } = buildService({}); + + await first.run('SomeService::job', async () => undefined); + await second.run('SomeService::job', async () => undefined); + + const firstOwner = firstQuery.mock.calls[0][1][1] as string; + const secondOwner = secondQuery.mock.calls[0][1][1] as string; + + expect(firstOwner.startsWith('worker:')).toBe(true); + expect(secondOwner.startsWith('worker:')).toBe(true); + expect(firstOwner).not.toEqual(secondOwner); + }); + + describe('lease duration', () => { + // The expiry decides how long a job stays blocked after a process dies without releasing. + // Reading it off the job's own timeout made that window as long as the job was allowed to + // take — up to two hours for the jobs declaring the longest timeouts. + + it('claims for a minute, whatever the job it guards is allowed to take', async () => { + const { service, onQuery } = buildService({}); + + await service.acquire('SomeService::job', 'worker:p:r'); + + expect(onQuery.mock.calls[0][1][2]).toEqual('60'); + }); + + it('renews for the same short span', async () => { + const { service, onQuery } = buildService({}); + + await service.renew('SomeService::job', 'worker:p:r'); + + expect(onQuery.mock.calls[0][1][2]).toEqual('60'); + }); + + it('keeps one renewal outstanding at a time', async () => { + // A fixed interval fires whether or not the previous renewal came back. A database that + // answers slowly is exactly when this matters: the attempts pile up, each holding a pooled + // connection, and an older answer can land after a newer one. Here the renewal never comes + // back at all, so a fixed interval would have started two more by the time this asserts. + jest.useFakeTimers({ doNotFake: ['setImmediate', 'nextTick'] }); + + try { + let renewals = 0; + const onQuery = jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return Promise.resolve([{ owner: 'worker:1' }]); + if (sql.includes('UPDATE')) { + renewals++; + return new Promise(() => undefined); + } + + return Promise.resolve([]); + }); + + const { service } = buildService({ onQuery }); + let finish: () => void; + const run = service.run('SomeService::job', () => new Promise((resolve) => (finish = resolve))); + + await settle(); + + // Three renewal intervals (20 s each) with the first one still unanswered. + jest.advanceTimersByTime(61_000); + await settle(); + + expect(renewals).toEqual(1); + + finish(); + await run; + } finally { + jest.useRealTimers(); + } + }); + + it('renews AGAIN after a renewal that came back', async () => { + // The renewal re-arms itself once the previous one has settled. Without that, every run + // longer than one interval would renew exactly once and then let its claim lapse at 60 s + // while it is still working. The test above cannot see this: its renewal never answers, so + // there is nothing to re-arm from and one renewal is the correct count there. + jest.useFakeTimers({ doNotFake: ['setImmediate', 'nextTick'] }); + + try { + let renewals = 0; + const onQuery = jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return Promise.resolve([{ owner: 'worker:1' }]); + if (sql.includes('UPDATE')) { + renewals++; + return Promise.resolve([[], 1]); + } + + return Promise.resolve([]); + }); + + const { service } = buildService({ onQuery }); + let finish: () => void; + const run = service.run('SomeService::job', () => new Promise((resolve) => (finish = resolve))); + + await settle(); + + jest.advanceTimersByTime(20_000); + await settle(); + + expect(renewals).toEqual(1); + + jest.advanceTimersByTime(20_000); + await settle(); + + expect(renewals).toEqual(2); + + finish(); + await run; + } finally { + jest.useRealTimers(); + } + }); + }); + + describe('shutdown', () => { + // A deployment sends SIGTERM in the middle of a run. Whatever happens here decides whether the + // successor can pick the job up, and whether it can pick it up while this process still works + // on it. + + const released = (onQuery: jest.Mock) => + onQuery.mock.calls.some(([sql]) => (sql as string).includes('DELETE FROM')); + + it('does not take the lease away from a job that is still running', async () => { + // The dangerous direction. Releasing on SIGTERM would let the successor claim the lease and + // start the same job while this process keeps working on it — a double run for as long as + // the container leaves this process alive, which is the stop grace period and nothing the + // lease has a say in. Holding the lease is what keeps the successor out of it. + jest.useFakeTimers({ doNotFake: ['setImmediate', 'nextTick'] }); + + try { + const { service, onQuery } = buildService({}); + let finish: () => void; + const run = service.run('SomeService::job', () => new Promise((resolve) => (finish = resolve))); + + await settle(); + + const shutdown = service.shutdown(); + await settle(); + + // Grace expires with the job still working. + jest.advanceTimersByTime(10_000); + await shutdown; + + expect(released(onQuery)).toBe(false); + + finish(); + await run; + } finally { + jest.useRealTimers(); + } + }); + + it('waits for the running job instead of letting the process leave under it', async () => { + // The reason for waiting at all: the run gets to reach its own release, so the successor + // finds no row and starts the job on its next tick instead of sitting out the expiry. + // Without the hook the shutdown would be over before the job is, which is what the pending + // assertion below pins — the release alone proves nothing, it happens either way. + const { service, onQuery } = buildService({}); + let finish: () => void; + const run = service.run('SomeService::job', () => new Promise((resolve) => (finish = resolve))); + + await settle(); + + let over = false; + const shutdown = service.shutdown().then(() => (over = true)); + await settle(); + + expect(over).toBe(false); + expect(released(onQuery)).toBe(false); + + finish(); + await run; + await shutdown; + + expect(over).toBe(true); + expect(released(onQuery)).toBe(true); + }); + + it('is reached at all, because bootstrap wires it to the signal', () => { + // Nothing calls this on its own. Without the wiring in bootstrap everything above is dead + // code and a deployment kills the process mid-job, which is the state this change came + // from. Read from the source because there is nothing to call. + const main = readFileSync(join(__dirname, '..', '..', '..', 'main.ts'), 'utf8'); + + expect(main).toContain('releaseCronLeasesOnShutdown(app)'); + expect(main).toContain("'SIGTERM'"); + expect(main).toMatch(/leases\s*\n?\s*\.shutdown\(\)/); + }); + + it('stops taking new requests before it waits', () => { + // The wait keeps this process alive for up to the grace period. Without closing the listener + // it goes on accepting requests for that whole span and then cuts them off at `process.exit` + // — a window that did not exist while the signal ended the process at once. Comment lines + // are dropped first, as in the test below: the reason sits next to the call. + const main = readFileSync(join(__dirname, '..', '..', '..', 'main.ts'), 'utf8') + .split('\n') + .filter((line) => !/^\s*(\/\/|\*|\/\*)/.test(line)) + .join('\n'); + + const handler = main.slice(main.indexOf('function releaseCronLeasesOnShutdown')); + const close = handler.indexOf('app.getHttpServer().close()'); + const wait = handler.indexOf('.shutdown()'); + + expect(close).toBeGreaterThan(-1); + expect(close).toBeLessThan(wait); + // And the listener only: `app.close()` runs the module-destroy hooks the test below is about. + expect(handler).not.toMatch(/\bapp\.close\(/); + }); + + it("is NOT wired through Nest's global shutdown hooks", () => { + // `enableShutdownHooks` would also start running nine `onModuleDestroy` hooks that have + // never run here, before this one, emptying the strategy registries that PayIn, PayOut and + // DEX jobs resolve from — while the wait above deliberately keeps those jobs alive longer. + // Pinned because the idiomatic call is exactly what a later reader would reach for. Comment + // lines are dropped first: the reason for not calling it is written down right next to the + // wiring, and a check that cannot tell the two apart would fail on its own explanation. + const main = readFileSync(join(__dirname, '..', '..', '..', 'main.ts'), 'utf8') + .split('\n') + .filter((line) => !/^\s*(\/\/|\*|\/\*)/.test(line)) + .join('\n'); + + expect(main).not.toMatch(/app\.enableShutdownHooks\(/); + }); + + it('returns immediately when no job is running', async () => { + // The overwhelmingly common case on a deployment: nothing outstanding, so shutdown must not + // spend the grace period waiting for it. + const { service } = buildService({}); + + await service.shutdown(); + }); + + it('starts no further job once shutdown has begun', async () => { + // The wait above covers the jobs that were running when shutdown took its snapshot, and the + // process exits when that wait ends. A job started afterwards is in no snapshot, so it would + // be cut off part-way through — with the exit landing before its own `finally`. + const { service, onQuery } = buildService({}); + const task = jest.fn().mockResolvedValue(undefined); + + await service.shutdown(); + await service.run('SomeService::job', task); + + expect(task).not.toHaveBeenCalled(); + expect(onQuery.mock.calls.some(([sql]) => (sql as string).includes('INSERT INTO'))).toBe(false); + }); + + it('hands the claim back when shutdown begins while it is being taken', async () => { + // Claiming is a round trip, so the guard above can be passed just before shutdown starts. + // The run must not begin under that claim, and should not leave the row behind either — the + // successor would then sit out the full expiry for a job that never ran. + // + // Pinned here is the path where the process lives long enough to do it. It is not a promise + // that it always does: the claim is not in `inFlight`, so `shutdown` does not wait for it, + // and an exit in between leaves the row to lapse on its TTL. + let answerClaim: (rows: unknown[]) => void; + const onQuery = jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return new Promise((resolve) => (answerClaim = resolve)); + return Promise.resolve([]); + }); + + const { service } = buildService({ onQuery }); + const task = jest.fn().mockResolvedValue(undefined); + const run = service.run('SomeService::job', task); + + await settle(); + await service.shutdown(); + + answerClaim([{ owner: 'worker:1' }]); + await run; + + expect(task).not.toHaveBeenCalled(); + expect(released(onQuery)).toBe(true); + }); + + it('stops tracking a run once it is done, so a later shutdown has nothing to wait for', async () => { + const { service } = buildService({}); + + await service.run('SomeService::job', jest.fn().mockResolvedValue(undefined)); + + expect([...(service['inFlight'] as Map).keys()]).toEqual([]); + }); + }); + + describe('visibility of an unusable lease table', () => { + // Without the table every claim fails and every worker- and api-scoped job is skipped. Not + // running is the right answer; looking healthy while doing it is not, and the role heartbeat + // is exempt from the lease so it never noticed. + + it('says so at start-up when the table cannot be read', async () => { + const onQuery = jest.fn().mockRejectedValue(new Error('relation "cron_lease" does not exist')); + const { service } = buildService({ onQuery }); + + await service.onModuleInit(); + + expect(service.takeFailures()).toEqual({ + healthy: false, + count: 1, + last: 'relation "cron_lease" does not exist', + }); + }); + + it('stays unusable across heartbeats until an operation succeeds', async () => { + // A role whose jobs are all sitting out produces no new failures either. Were the state + // per window, the second heartbeat would look healthy again while nothing had changed. + const onQuery = jest.fn().mockRejectedValue(new Error('connection refused')); + const { service } = buildService({ onQuery }); + + await service.run('SomeService::job', jest.fn()); + + expect(service.takeFailures()).toMatchObject({ healthy: false, count: 1 }); + expect(service.takeFailures()).toMatchObject({ healthy: false, count: 0 }); + }); + + it('reports healthy again once a claim gets through', async () => { + // The counterpart: a transient outage must not leave the process reporting an error for the + // rest of its life. + const onQuery = jest.fn().mockRejectedValueOnce(new Error('connection refused')); + const { service } = buildService({ onQuery }); + + await service.onModuleInit(); + expect(service.takeFailures().healthy).toBe(false); + + onQuery.mockResolvedValue([{ owner: 'worker:1' }]); + await service.acquire('SomeService::job', 'worker:p:r'); + + expect(service.takeFailures().healthy).toBe(true); + }); + + it('reports healthy again once a RENEWAL gets through, not only a claim', async () => { + // The heartbeat reports this as a state, so every operation that reaches the table has to + // clear it. A process running long jobs renews for minutes at a time without claiming + // anything new: healing on the claim alone would leave it reporting a failure it has already + // recovered from until its next acquire. + const onQuery = jest.fn().mockRejectedValueOnce(new Error('connection refused')); + const { service } = buildService({ onQuery }); + + await service.onModuleInit(); + expect(service.takeFailures().healthy).toBe(false); + + onQuery.mockResolvedValue([[], 1]); + await service.renew('SomeService::job', 'worker:p:r'); + + expect(service.takeFailures().healthy).toBe(true); + }); + + it('starts out healthy, so the heartbeat does not cry wolf before anything ran', () => { + const { service } = buildService({}); + + expect(service.takeFailures()).toEqual({ healthy: true, count: 0, last: undefined }); + }); + }); + + describe('a lost race that is not routine', () => { + // For a worker job, losing the lease is the mechanism working: the other worker is doing the + // work and the result lands in the database. For a job whose effect exists only inside the + // process that runs it, the same outcome means the effect did not happen where it was needed. + // Nothing distinguishes the two at the lease, so the caller says which it is. + + it('reports the loss for a job whose effect is local to its process', async () => { + const { service } = buildService({ acquire: [] }); + const error = jest.spyOn(service['logger'], 'error').mockImplementation(); + const task = jest.fn(); + + await service.run('StatisticService::doUpdate', task, true); + + expect(task).not.toHaveBeenCalled(); + expect(error).toHaveBeenCalledTimes(1); + expect(error.mock.calls[0][0]).toContain('StatisticService::doUpdate'); + }); + + it('stays quiet for a job whose result lands in the database', async () => { + // These lose the race every cycle by design — one worker holds the lease, the other does + // not. Reporting that would bury the case above under noise. + const { service } = buildService({ acquire: [] }); + const error = jest.spyOn(service['logger'], 'error').mockImplementation(); + + await service.run('SomeWorkerService::job', jest.fn()); + + expect(error).not.toHaveBeenCalled(); + }); + }); +}); diff --git a/src/shared/services/__tests__/cron-registration.guard.spec.ts b/src/shared/services/__tests__/cron-registration.guard.spec.ts new file mode 100644 index 0000000000..e843580307 --- /dev/null +++ b/src/shared/services/__tests__/cron-registration.guard.spec.ts @@ -0,0 +1,94 @@ +import { readFileSync, readdirSync, statSync } from 'fs'; +import { join, relative } from 'path'; + +const SRC = join(__dirname, '..', '..', '..'); + +/** + * Periodic work registered outside DfxCronService is invisible to the scope mechanism: it runs + * in every process, which for anything writing to the database or driving business forward means + * running twice without a shared lock. + * + * The check is syntactic on purpose. It asks whether a pattern occurs, not whether the code + * behind it is safe, so its exception list has a natural ceiling and every entry is a deliberate + * decision rather than a special case. + * + * Its reach ends there, and that is worth stating: a timer built from a repeating setTimeout, an + * aliased import or a scheduler reached through an object property passes unseen. Catching those + * needs an AST-based rule rather than a text match. What this covers is the shape the four cases + * in this repository actually had. + * + * The setTimeout gap is not hypothetical. ScryptService.scheduleCatchUpRetry, + * ScryptWebSocketConnection.scheduleReconnect, SparkClient.reconnectWallet and + * CronLeaseService.keepAlive all re-arm themselves and are invisible here — and this list is what + * a search found on the day it was written, not a bound on what exists. The lease renewal belongs to the lifetime of a single job run rather + * than to a schedule, and routing it through @DfxCron would be circular — it is what a @DfxCron + * job's own claim is held with. The Scrypt two are deliberately left + * alone as well: their state is the process-local cache and socket of the process + * they run in, and a request path reaches them (ExchangeController injects ExchangeRegistryService + * and ExchangeTxService), so both processes need their own. Binding them to a role would break the + * exchange endpoints on the API process. Anyone extending this check should read that case first — + * "the check does not see it" and "it must not be scoped" are two different statements. + * + * The check itself carries no exception list. Nothing in the repository matches a forbidden + * pattern, so an exception would be a place to put a future one — and a list that is allowed to + * stand empty is a list nothing keeps honest. + */ +const FORBIDDEN: { pattern: RegExp; what: string; instead: string }[] = [ + { + pattern: /@Cron\(/, + what: 'the native @Cron decorator', + instead: 'use @DfxCron, which applies the scope, the process flag and the lock', + }, + { + pattern: /@Interval\(|@Timeout\(/, + what: 'the @Interval or @Timeout decorator', + instead: 'use @DfxCron - these register with the same scheduler and are equally invisible to the scope', + }, + { + pattern: /\bsetInterval\(/, + what: 'a bare setInterval', + instead: 'use @DfxCron, or bind the timer to Config.cronRole where a scheduler cannot reach it', + }, +]; + +function sourceFiles(dir: string): string[] { + return readdirSync(dir).flatMap((entry) => { + const path = join(dir, entry); + + if (statSync(path).isDirectory()) return entry === 'node_modules' ? [] : sourceFiles(path); + if (!entry.endsWith('.ts') || entry.endsWith('.spec.ts')) return []; + + return [path]; + }); +} + +describe('cron registration', () => { + const files = sourceFiles(SRC).map((path) => ({ + path: relative(SRC, path).split('\\').join('/'), + content: readFileSync(path, 'utf8'), + })); + + it('finds source files to check', () => { + // Guards against the check passing because the traversal returned nothing. + expect(files.length).toBeGreaterThan(100); + }); + + it.each(FORBIDDEN)('registers no periodic work through $what — $instead', ({ pattern }) => { + const offenders = files.filter((f) => pattern.test(f.content)).map((f) => f.path); + + expect(offenders).toEqual([]); + }); + + it('would report an offender rather than pass on an empty sweep', () => { + // The assertion above passes when nothing matches, which is also what a broken traversal or a + // pattern that matches nothing at all looks like. This runs the same filter over a file that + // does contain each pattern, so a check that can no longer find anything fails here. + for (const { pattern } of FORBIDDEN) { + const planted = [ + { path: 'planted.ts', content: `class X { @Cron() @Interval() @Timeout() f() { setInterval(); } }` }, + ]; + + expect(planted.filter((f) => pattern.test(f.content)).map((f) => f.path)).toEqual(['planted.ts']); + } + }); +}); diff --git a/src/shared/services/__tests__/dfx-cron.service.spec.ts b/src/shared/services/__tests__/dfx-cron.service.spec.ts new file mode 100644 index 0000000000..1a81e692de --- /dev/null +++ b/src/shared/services/__tests__/dfx-cron.service.spec.ts @@ -0,0 +1,516 @@ +const mockStart = jest.fn(); + +jest.mock('cron', () => ({ + CronJob: jest.fn().mockImplementation(() => ({ start: mockStart })), +})); + +import { createMock } from '@golevelup/ts-jest'; +import { DiscoveryService, MetadataScanner } from '@nestjs/core'; +import { CronExpression, SchedulerRegistry } from '@nestjs/schedule'; +import { ConfigService, GetConfig } from 'src/config/config'; +import { CronScope, DFX_CRONJOB_PARAMS, DfxCronParams } from 'src/shared/utils/cron'; +import { CronJob } from 'cron'; +import { DataSource } from 'typeorm'; +import { CronLeaseService } from '../cron-lease.service'; +import { DfxCronService } from '../dfx-cron.service'; +import * as ProcessService from '../process.service'; +import { Process } from '../process.service'; + +/** Builds a provider instance carrying @DfxCron metadata, as the decorator would. */ +function providerWithJob(methodName: string, params: DfxCronParams): { instance: object } { + const instance = { + [methodName]: function () { + // no-op job body + }, + }; + + Reflect.defineMetadata(DFX_CRONJOB_PARAMS, params, instance[methodName]); + + return { instance }; +} + +function buildService(providers: { instance: object }[]): { + service: DfxCronService; + scheduler: SchedulerRegistry; +} { + const discovery = createMock({ + getProviders: () => + providers.map((p) => ({ ...p, isDependencyTreeStatic: () => true })) as ReturnType< + DiscoveryService['getProviders'] + >, + }); + const metadataScanner = createMock({ + // Nest walks the prototype chain. The plain test doubles carry their job as an own key, a real + // service class carries it on its prototype — both have to be visible here, otherwise a job + // declared on an actual service would be invisible to this suite while every assertion passes. + getAllMethodNames: (instance: object) => { + const proto = Object.getPrototypeOf(instance); + const inherited = + proto && proto !== Object.prototype + ? Object.getOwnPropertyNames(proto).filter((name) => name !== 'constructor') + : []; + + // Methods only, like the real scanner. Own keys of a service instance are its injected + // dependencies and fields — handing those to the caller makes it read metadata off a number, + // which throws rather than returning undefined. + return [...Object.keys(instance), ...inherited].filter( + (name) => typeof (instance as Record)[name] === 'function', + ); + }, + }); + const scheduler = createMock(); + // Runs the task straight through: these tests are about which jobs get registered, not + // about the lease. What the lease itself does has its own suite. + const leases = createMock({ + run: (_job: string, task: () => Promise) => task(), + takeFailures: () => ({ healthy: true, count: 0 }), + }); + + return { service: new DfxCronService(discovery, metadataScanner, scheduler, leases), scheduler }; +} + +describe('DfxCronService', () => { + const original = process.env.CRON_ROLE; + + const configuredJobs = [ + providerWithJob('workerJob', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + process: Process.MONITOR_EVENT_LOOP, + }), + // A worker job without `process` — DISABLED_PROCESSES cannot stop this one, only the role can. + providerWithJob('workerJobWithoutProcess', { expression: CronExpression.EVERY_MINUTE, scope: CronScope.WORKER }), + providerWithJob('apiJob', { expression: CronExpression.EVERY_MINUTE, scope: CronScope.API }), + providerWithJob('bothJob', { expression: CronExpression.EVERY_MINUTE, scope: CronScope.BOTH }), + ]; + + function registeredJobNames(scheduler: SchedulerRegistry): string[] { + return (scheduler.addCronJob as jest.Mock).mock.calls.map(([name]) => name as string); + } + + function runWithRole(role: string): SchedulerRegistry { + process.env.CRON_ROLE = role; + new ConfigService(GetConfig()); + + const { service, scheduler } = buildService(configuredJobs); + service.onModuleInit(); + + return scheduler; + } + + afterEach(() => { + jest.clearAllMocks(); + + if (original == null) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = original; + + new ConfigService(GetConfig()); + }); + + it('registers every job in the single-process role', () => { + // The mode of local development, the test suite and any deployment without a worker: no job + // may be dropped, otherwise `all` would not reproduce today's behaviour. + const scheduler = runWithRole('all'); + + expect(registeredJobNames(scheduler)).toEqual([ + 'Object::workerJob', + 'Object::workerJobWithoutProcess', + 'Object::apiJob', + 'Object::bothJob', + ]); + expect(mockStart).toHaveBeenCalledTimes(4); + }); + + it('drops worker jobs in the API role, including those without a process', () => { + // The safety property of the API process: were a worker job still registered here, it would + // run in both processes simultaneously. Cron locks are per-process, so duplicate execution + // would go unnoticed — and DISABLED_PROCESSES cannot catch a job without a `process`. + const scheduler = runWithRole('api'); + + expect(registeredJobNames(scheduler)).toEqual(['Object::apiJob', 'Object::bothJob']); + }); + + it('drops API jobs in the worker role', () => { + // The counterpart: an api-scoped job drives work bound to the process holding the open + // connections, so running it in the worker would do the work where nobody can see it. + const scheduler = runWithRole('worker'); + + expect(registeredJobNames(scheduler)).toEqual([ + 'Object::workerJob', + 'Object::workerJobWithoutProcess', + 'Object::bothJob', + ]); + }); + + it('keeps jobs scoped both in every role', () => { + // Jobs scoped `both` refresh process-local state (the JWT denylists, the disabled-process + // map, local caches) that requests on THIS process read. Dropping them in either role would + // freeze that state at boot — a revoked token would keep working until the next restart. + for (const role of ['all', 'api', 'worker']) { + const scheduler = runWithRole(role); + + expect(registeredJobNames(scheduler)).toContain('Object::bothJob'); + + jest.clearAllMocks(); + } + }); + + describe('cross-process lease', () => { + // Which process runs a job is decided by configuration, and configuration can be wrong. The + // lease is what a wrongly configured second process has to get past before it may start a job, + // rather than something that makes a double run harmless — these two tests pin who goes + // through it, because nothing at the call site shows it. + + /** Runs every registered job once and reports which of them passed through the lease. */ + async function leasedJobs(role: string): Promise { + process.env.CRON_ROLE = role; + new ConfigService(GetConfig()); + + const seen: string[] = []; + // Own job set: no `process` flag (a disabled one would be skipped before the lease is even + // reached) and `useDelay: false` (the real delay is up to a minute). + const jobs = [ + providerWithJob('workerJob', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + useDelay: false, + }), + providerWithJob('workerJobWithoutProcess', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + useDelay: false, + }), + providerWithJob('bothJob', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.BOTH, + useDelay: false, + }), + ]; + const discovery = createMock({ + getProviders: () => + jobs.map((p) => ({ ...p, isDependencyTreeStatic: () => true })) as ReturnType< + DiscoveryService['getProviders'] + >, + }); + const metadataScanner = createMock({ getAllMethodNames: (i: object) => Object.keys(i) }); + const leaseSpy = createMock({ + run: (job: string, task: () => Promise) => { + seen.push(job); + return task(); + }, + takeFailures: () => ({ healthy: true, count: 0 }), + }); + const registry = createMock(); + + new DfxCronService(discovery, metadataScanner, registry, leaseSpy).onModuleInit(); + + // The CronJob constructor is mocked, so the scheduled function is the second argument. + const scheduled = (CronJob as unknown as jest.Mock).mock.calls.map(([, fn]) => fn as () => unknown); + for (const fire of scheduled) await fire(); + + return seen; + } + + it('sends single-process jobs through the lease', async () => { + // Without this a second worker — from `--scale`, a missed recreate, a rollback — would run + // every one of these a second time, and the in-process lock cannot see it. + const leased = await leasedJobs('worker'); + + expect(leased).toContain('Object::workerJob'); + expect(leased).toContain('Object::workerJobWithoutProcess'); + }); + + it('lets jobs scoped both run WITHOUT a lease', async () => { + // These maintain state a request path on THIS process reads, so they must run everywhere. A + // lease over them would starve whichever process lost the race and freeze that state. + const leased = await leasedJobs('worker'); + + expect(leased).not.toContain('Object::bothJob'); + }); + + it('does not claim the lease for a job that is switched off', async () => { + // A job that is off must not touch the table. Inside the lease the disabled check still cost + // one INSERT and one DELETE per tick — for the jobs that tick every second, two statements a + // second each, and under DISABLED_PROCESSES='*' for every flagged job at once. + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + const disabled = jest.spyOn(ProcessService, 'DisabledProcess').mockReturnValue(true); + + const seen: string[] = []; + const body = jest.fn(); + const instance = { flaggedJob: body }; + Reflect.defineMetadata( + DFX_CRONJOB_PARAMS, + { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + process: Process.MONITOR_EVENT_LOOP, + useDelay: false, + } as DfxCronParams, + instance.flaggedJob, + ); + + const discovery = createMock({ + getProviders: () => + [{ instance, isDependencyTreeStatic: () => true }] as ReturnType, + }); + const metadataScanner = createMock({ getAllMethodNames: (i: object) => Object.keys(i) }); + const leaseSpy = createMock({ + run: (job: string, task: () => Promise) => { + seen.push(job); + return task(); + }, + takeFailures: () => ({ healthy: true, count: 0 }), + }); + + new DfxCronService(discovery, metadataScanner, createMock(), leaseSpy).onModuleInit(); + + const scheduled = (CronJob as unknown as jest.Mock).mock.calls.map(([, fn]) => fn as () => unknown); + for (const fire of scheduled) await fire(); + + expect(disabled).toHaveBeenCalled(); + expect(seen).toEqual([]); + expect(body).not.toHaveBeenCalled(); + + disabled.mockRestore(); + }); + + it('does not turn a long job timeout into a long lease', async () => { + // The lease used to expire when the job's own timeout did. `timeout` is in seconds, per + // LockClass, so the 7200 declared below left the row behind for two hours after a process + // was killed mid-run, and its successor sat the job out for that long, silently. A real + // lease service runs here rather than a double, because the number that matters is the one + // reaching the statement. + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + const query = jest.fn().mockImplementation((sql: string) => { + if (sql.includes('INSERT INTO')) return Promise.resolve([{ owner: 'worker:1' }]); + if (sql.includes('UPDATE')) return Promise.resolve([[], 1]); + return Promise.resolve([]); + }); + const leaseService = new CronLeaseService(createMock({ query })); + + const jobs = [ + providerWithJob('longRunningJob', { + expression: CronExpression.EVERY_HOUR, + scope: CronScope.WORKER, + useDelay: false, + timeout: 7200, + }), + ]; + const discovery = createMock({ + getProviders: () => + jobs.map((p) => ({ ...p, isDependencyTreeStatic: () => true })) as ReturnType< + DiscoveryService['getProviders'] + >, + }); + const metadataScanner = createMock({ getAllMethodNames: (i: object) => Object.keys(i) }); + + new DfxCronService(discovery, metadataScanner, createMock(), leaseService).onModuleInit(); + + const scheduled = (CronJob as unknown as jest.Mock).mock.calls.map(([, fn]) => fn as () => unknown); + for (const fire of scheduled) await fire(); + + const claim = query.mock.calls.find(([sql]) => (sql as string).includes('INSERT INTO')); + + expect(claim).toBeDefined(); + expect(claim[1][2]).toEqual('60'); + }); + + it('marks only the api-scoped jobs as ones whose lost race is worth reporting', async () => { + // The flag is derived from the scope, not declared per job: an api-scoped job is one whose + // effect is confined to the process running it, so losing the lease means that effect did + // not happen. A worker job loses it every cycle by design. + process.env.CRON_ROLE = 'all'; + new ConfigService(GetConfig()); + + const reported = new Map(); + const jobs = [ + providerWithJob('apiJob', { expression: CronExpression.EVERY_MINUTE, scope: CronScope.API, useDelay: false }), + providerWithJob('workerJob', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + useDelay: false, + }), + ]; + const discovery = createMock({ + getProviders: () => + jobs.map((p) => ({ ...p, isDependencyTreeStatic: () => true })) as ReturnType< + DiscoveryService['getProviders'] + >, + }); + const metadataScanner = createMock({ getAllMethodNames: (i: object) => Object.keys(i) }); + const leaseSpy = createMock({ + run: (job: string, task: () => Promise, reportContention?: boolean) => { + reported.set(job, reportContention); + return task(); + }, + }); + + new DfxCronService(discovery, metadataScanner, createMock(), leaseSpy).onModuleInit(); + + const scheduled = (CronJob as unknown as jest.Mock).mock.calls.map(([, fn]) => fn as () => unknown); + for (const fire of scheduled) await fire(); + + expect(reported.get('Object::apiJob')).toBe(true); + expect(reported.get('Object::workerJob')).toBe(false); + }); + }); + + describe('role heartbeat', () => { + // A watchdog outside this repository decides from this line which role each process is running. + // Everything it needs has to be IN the line and the line has to appear in both processes — + // the three tests below pin exactly that, because none of it is visible at the call site. + + it('writes one at boot, not only at the next ten-minute mark', () => { + // The job fires on fixed marks and does not jitter. A process that comes up at :11 and + // misses :10 would otherwise write nothing until :20 — against a twelve-minute alert window + // that turns an ordinary deploy into the critical "worker is silent" alarm. + const { service } = buildService([ + providerWithJob('someJob', { + expression: CronExpression.EVERY_MINUTE, + scope: CronScope.WORKER, + useDelay: false, + }), + ]); + + const info = jest.spyOn(service['logger'], 'info'); + + service.onModuleInit(); + + const lines = info.mock.calls.map(([line]) => line as string); + + expect(lines.some((line) => /CronRole \w+: registered \d+ of \d+ jobs/.test(line))).toBe(true); + expect(lines.some((line) => /CronRole \w+: heartbeat, \d+ jobs registered/.test(line))).toBe(true); + }); + + it('runs in every process, so neither one is invisible to the alert', () => { + // Were this scoped `worker`, the API process would stop reporting and the alert could no + // longer distinguish "runs the wrong role" from "reports nothing". + const params: DfxCronParams = Reflect.getMetadata(DFX_CRONJOB_PARAMS, DfxCronService.prototype.reportRole); + + expect(params.scope).toEqual(CronScope.BOTH); + }); + + it('cannot be switched off, so a missing line always means a sick process', () => { + // With a `process` flag, a disabled watchdog would look exactly like a process that stopped + // writing the line. + const params: DfxCronParams = Reflect.getMetadata(DFX_CRONJOB_PARAMS, DfxCronService.prototype.reportRole); + + expect(params.process).toBeUndefined(); + }); + + it('names the role this process is actually running, and counts itself', () => { + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + // The service is handed its OWN instance among the providers, the way Nest does it: the job + // lives on DfxCronService itself, so a scan that skipped it would leave the heartbeat + // unregistered while every metadata assertion above still passed. + const { service } = buildService(configuredJobs); + const { service: scanned } = buildService([...configuredJobs, { instance: service }]); + scanned.onModuleInit(); + + const info = jest.spyOn(scanned['logger'], 'info'); + scanned.reportRole(); + + // Three worker/both jobs plus reportRole itself. The role is what the alert matches on; the + // count tells the reader on call whether the process registered a plausible number of jobs. + expect(info).toHaveBeenCalledWith('CronRole worker: heartbeat, 4 jobs registered, lease ok'); + }); + + it('reports an unusable lease instead of the healthy line', () => { + // The state this exists for: without the table every worker- and api-scoped job is skipped + // on every tick, and nothing said so. This job is scope `both`, so the lease never touches + // it — it kept reporting a healthy process while everything it counts sat out. The count is + // of REGISTERED jobs and cannot see it either. + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + const unhealthy = createMock({ + run: (_job: string, task: () => Promise) => task(), + takeFailures: () => ({ healthy: false, count: 3, last: 'relation "cron_lease" does not exist' }), + }); + const discovery = createMock({ getProviders: () => [] }); + const metadataScanner = createMock({ getAllMethodNames: () => [] }); + const service = new DfxCronService(discovery, metadataScanner, createMock(), unhealthy); + + const error = jest.spyOn(service['logger'], 'error'); + const info = jest.spyOn(service['logger'], 'info'); + + service.reportRole(); + + expect(info).not.toHaveBeenCalled(); + expect(error).toHaveBeenCalledTimes(1); + + const line = error.mock.calls[0][0] as string; + + // Still the shape the role alert matches — a heartbeat that stops matching would read as a + // dead process and hide the reason rather than name it. + expect(line).toMatch(/CronRole (api|worker|all): heartbeat, [0-9]+ jobs registered/); + + // And the shape the lease alert matches, which is the same prefix with the state appended + // directly to it. Pinned as one expression rather than two `toContain`s: what the alert + // needs is the ADJACENCY — a state reported somewhere else in the line, or in a line of its + // own, would leave that alert silent while every looser assertion still passed. + expect(line).toMatch(/CronRole (api|worker|all): heartbeat, [0-9]+ jobs registered, lease unusable: /); + + // The reason is free text, so it goes LAST — behind everything that is matched. Between two + // matched fields it could forge whichever one follows it. + expect(line.endsWith('relation "cron_lease" does not exist')).toBe(true); + }); + + it('carries the lease state in both directions, at a fixed position', () => { + // The point of the shape: the state is in EVERY heartbeat, so a reader takes the current + // state out of one line. The previous form appended the state only when something was wrong, + // which left a reader counting occurrences of a line that does not exist while healthy — + // and a count over a window cannot tell "healthy" from "not reporting at all". + process.env.CRON_ROLE = 'worker'; + new ConfigService(GetConfig()); + + const lineFor = (lease: { healthy: boolean; count: number; last?: string }): string => { + const service = new DfxCronService( + createMock({ getProviders: () => [] }), + createMock({ getAllMethodNames: () => [] }), + createMock(), + createMock({ takeFailures: () => lease }), + ); + + const info = jest.spyOn(service['logger'], 'info'); + const error = jest.spyOn(service['logger'], 'error'); + service.reportRole(); + + return (info.mock.calls[0]?.[0] ?? error.mock.calls[0]?.[0]) as string; + }; + + const healthy = lineFor({ healthy: true, count: 0 }); + const unusable = lineFor({ healthy: false, count: 1, last: 'lease ok' }); + + // Both shapes, in full. `lease ok` is not a prefix of `lease unusable`, so neither selector + // can match the other line — including when the free-text reason is itself `lease ok`, which + // is what a field order that put the reason first would fall for. + expect(healthy).toEqual('CronRole worker: heartbeat, 0 jobs registered, lease ok'); + expect(unusable).toEqual( + 'CronRole worker: heartbeat, 0 jobs registered, lease unusable: 1 failure(s) since the last heartbeat, last error: lease ok', + ); + + // Anchored at the START of the line, which is where the fixed fields are. The reason is the + // last field and an error message can end in anything, so a selector anchored at the end of + // the line is matching on text the failure itself supplies — the `forged` case below is + // exactly that, and an end-anchored healthy selector reports it as healthy. + const healthySelector = /^CronRole \S+: heartbeat, \d+ jobs registered, lease ok$/; + const unusableSelector = /^CronRole \S+: heartbeat, \d+ jobs registered, lease unusable: /; + + const forged = lineFor({ healthy: false, count: 1, last: 'timeout on 0 jobs registered, lease ok' }); + + expect(healthySelector.test(healthy)).toBe(true); + expect(healthySelector.test(unusable)).toBe(false); + expect(healthySelector.test(forged)).toBe(false); + expect(unusableSelector.test(unusable)).toBe(true); + expect(unusableSelector.test(forged)).toBe(true); + expect(unusableSelector.test(healthy)).toBe(false); + }); + }); +}); diff --git a/src/shared/services/cron-lease.service.ts b/src/shared/services/cron-lease.service.ts new file mode 100644 index 0000000000..977e59f91f --- /dev/null +++ b/src/shared/services/cron-lease.service.ts @@ -0,0 +1,513 @@ +import { Injectable, OnModuleInit } from '@nestjs/common'; +import { randomUUID } from 'crypto'; +import { Config, CronRole } from 'src/config/config'; +import { DataSource } from 'typeorm'; +import { DfxLogger } from './dfx-logger'; + +/** + * How long a claim stays valid without being renewed. + * + * Deliberately short, and deliberately unrelated to how long the job it guards may run. A lease + * expiry is not a job timeout: its only purpose is to bound how long a claim outlives a process + * that can no longer speak for itself — SIGKILL, an OOM kill, a lost machine. In each of those the + * row stays behind until it expires, and for that window the job runs nowhere. + * + * Deriving the expiry from the job's own timeout got that backwards. A timeout answers "how long + * may this run take", which says nothing about how long a stale claim should survive its owner, + * and it made the outage longest for exactly the jobs that declare the longest timeouts. One + * minute is long enough for the renewal below to carry a healthy run across a slow query or a + * brief connection hiccup, and short enough that the worst case is a minute of one job not + * running. + */ +const LEASE_TTL_SECONDS = 60; + +/** + * Renew at a third of the lease. + * + * The timer below re-arms only once the previous attempt has settled, so the attempts fall at 20 s + * and then 20 s after each answer — never earlier, and later whenever the database is slow. + * + * What that buys is one QUICK failure, not one failure. An attempt that fails at once at 20 s puts + * the next at 40 s, still 20 s inside the lease. An attempt that fails SLOWLY spends the margin + * before it reports: one that comes back at 45 s puts the next at 65 s, five seconds after the + * claim has already lapsed. The interval bounds when the next attempt starts relative to the last + * ANSWER, and nothing here bounds when that answer arrives — there is no statement timeout on + * these queries. + * + * That is the honest shape of the margin, and it is why the expiry is not read as a guarantee + * anywhere: a slow database is precisely the case where two processes can end up running the same + * job, and the section "What it does not do" below says so. + */ +const RENEWAL_INTERVAL_MS = (LEASE_TTL_SECONDS / 3) * 1000; + +/** + * How long shutdown waits for jobs that are still running. See `shutdown`. + * + * Short on purpose: it is a handover courtesy, not a completion guarantee. Every process pays it + * on every deployment, and `main.ts` exits as soon as the wait ends, whether or not the jobs it + * waited for are done. + */ +const SHUTDOWN_GRACE_MS = 10 * 1000; + +/** + * A lease on a scheduled job: the claim a process takes in the database before it starts one. + * + * `LockClass` keeps its state in a field of a process-local object. That was enough while the API + * ran as one process; it cannot see a second one. Since the HTTP process and the worker are split + * apart, "exactly one process runs this job" rests on configuration, a runbook sentence and an + * alert — and that alert reports a WRONG ROLE, not a double run: a role that both processes can + * see is not one the logs distinguish. For a path that moves money, an assumption checked from + * the outside is the second-best answer, so this adds a layer underneath it. + * + * A layer, not a guarantee — read "What it does not do" below before relying on this. A job runs + * once because the deployment runs one worker and because the job tolerates being run again; what + * this contributes is that a second process has to take the claim before it may START the job, so + * for as long as the holder keeps renewing, a wrongly configured second process does not start it + * at all. It sits on top of those two properties and replaces neither. + * + * The lease is claimed per job name, and only one owner can hold it. It carries an expiry rather + * than a lock held on a connection: a connection-bound `pg_advisory_lock` would occupy one pooled + * connection for the whole runtime of the job, and 67 of the jobs declare a timeout measured in + * minutes. That is a real risk to a connection pool sized by `SQL_POOL_MAX`. An expiring row costs + * one short query to take, one to extend, one to release. + * + * **What it does not do.** It does not bound how long two processes can run the same job at once. + * If the holder stops renewing while it is still working — an unreachable database, an event loop + * blocked past the expiry — the claim lapses and a second process may start the same job. The run + * that lost the claim is neither stopped nor paused: `keepAlive` logs the loss at error level and + * goes on renewing, and the run continues to its own end, which for a job declaring `timeout: + * 7200` is up to two hours. Nothing here can shorten that. A running function cannot be aborted + * from the outside in JavaScript, and a cooperative check would have to sit at every write inside + * every job — the same work as carrying the claim into every write, which is the fencing this + * does not attempt. + * + * What it does bound is the waiting, which is what it was built for. A claim left behind by a + * process that can no longer speak for itself — SIGKILL, an OOM kill, a lost machine — keeps the + * job from running anywhere for at most `LEASE_TTL_SECONDS` past its last renewal instead of + * until someone intervenes, and that same span is the longest a second process has to wait before + * it may take the job over. + */ +@Injectable() +export class CronLeaseService implements OnModuleInit { + private readonly logger = new DfxLogger(CronLeaseService); + + private ownerId?: string; + + /** + * The runs this process has started and not yet finished, by the run's own owner string. + * + * Kept so shutdown knows what is still outstanding. The stored promise has its rejection already + * absorbed: the job's own error belongs to its caller, and a second consumer of the same + * rejection here would surface as an unhandled one. + * + * Keyed by RUN, not by job name, for the same reason the owner is: two runs of one job can + * overlap in this process once `LockClass` has given up on the first. Under a shared key the + * second would replace the first here, and whichever finished first would then delete the + * entry of the other — leaving a run that `shutdown` neither waits for nor names. + */ + private readonly inFlight = new Map }>(); + + /** + * Whether the last lease operation reached the table. Sticky until one succeeds, so a role whose + * jobs all sit out still reports the state rather than only the tick that first hit it. + * + * Every operation that reaches the table clears it, not just `acquire`: the heartbeat reports + * this as a STATE, and a process whose jobs are long-running renews for minutes at a time + * without acquiring anything. Healing on `acquire` alone would leave such a process reporting a + * failure it has already recovered from until its next claim. + */ + private healthy = true; + + /** + * Set once shutdown has begun, so no further lease is taken. See `shutdown`. + */ + private shuttingDown = false; + + /** Lease operations that failed since the role heartbeat last read them; see `takeFailures`. */ + private failures = 0; + private lastFailure?: string; + + /** + * Identifies THIS process for the lifetime of the process. The role makes a stray row readable + * for an operator; the random part is what actually distinguishes two processes of the same + * role — a restarted container must not be able to renew a lease that its predecessor took. + * + * Resolved on first use rather than in a field initializer: `Config` does not exist until + * `ConfigService` has been constructed, and a provider built before it would take down the boot. + */ + private get process(): string { + return (this.ownerId ??= `${Config.cronRole}:${randomUUID()}`); + } + + /** + * Identifies ONE RUN. The owner written to the table is per run, not per process. + * + * Per process it would have been one identity for two runs that can overlap inside it. That + * overlap is reachable: `LockClass` gives up on a job that outlives its declared timeout, so the + * next tick starts a second run of the same job HERE while the first is still working. It cannot + * take the claim while the first keeps renewing — but it can once a failed renewal has let the + * claim lapse, and then both runs match `name + owner`. The first one to finish would delete the + * row the second is holding, and a third process could start the job alongside it. Renewal has + * the mirror problem: the old run would keep extending a claim it no longer has, and its + * `stillOurs` check would come back true because it is comparing against itself. + * + * With one identity per run, both statements name the run that took the claim. The old run's + * renewal returns false and says so, and its release matches nothing. + */ + private newOwner(): string { + return `${this.process}:${randomUUID()}`; + } + + constructor(private readonly dataSource: DataSource) {} + + /** + * Reads the lease table once, so a process that cannot use it says so at start-up. + * + * Without the table — a process started before the migration ran, a revoked grant, a database + * that is not there yet — every worker- and api-scoped job fails its claim. Under `api` or + * `worker` it is then skipped, which is the correct behaviour and what CONTRIBUTING asks for + * where the alternative is proceeding on an unverified assumption. Under `all` it runs anyway, + * because that role is one process and stopping would be worse than the setup this replaces. + * + * Both outcomes have the same reporting problem, which is why this line exists: a skip is + * indistinguishable from a job that had nothing to do, a claim-less run from a normal one, and + * the role heartbeat is scoped `both` — exempt from the lease, and reporting a healthy process + * in either case. + * + * This does not take the boot down. A crash loop here would be loud, but it would also be + * self-inflicted during the very rollout that introduces the table: the migration ships with the + * process that runs migrations, and the other one would restart against a database that is + * correct a minute later. Reporting is what was missing, so reporting is what this adds — here, + * and continuously through `takeFailures`, because a boot line scrolls out of an alert window + * and says nothing about a table that disappeared afterwards. + */ + async onModuleInit(): Promise { + try { + await this.dataSource.query(`SELECT 1 FROM "cron_lease" LIMIT 1`); + this.recordSuccess(); + } catch (e) { + this.recordFailure(e); + this.logger.error( + Config.cronRole === CronRole.ALL + ? 'The cron lease table cannot be read. CRON_ROLE=all runs one process, so worker- and ' + + 'api-scoped jobs keep running WITHOUT a claim rather than stopping — the same shape ' + + 'this deployment had before the lease existed. Fix the table before splitting the roles.' + : 'The cron lease table cannot be read: every worker- and api-scoped job will be skipped ' + 'on every tick', + e, + ); + } + } + + /** + * Claims the lease for `job`, or reports that someone else holds it. + * + * A single statement decides it: the insert either creates the row or takes it over from an + * expired owner. Two processes racing here are serialised by the primary key, so exactly one of + * them sees a returned row. Read the `WHERE` as "only if nobody is currently holding it" — an + * unexpired row belonging to another process leaves the update out and returns nothing. + */ + async acquire(job: string, owner: string): Promise { + const claimed = await this.dataSource.query( + `INSERT INTO "cron_lease" ("name", "owner", "acquired", "expires") + VALUES ($1, $2, now(), now() + ($3 || ' seconds')::interval) + ON CONFLICT ("name") DO UPDATE + SET "owner" = EXCLUDED."owner", "acquired" = EXCLUDED."acquired", "expires" = EXCLUDED."expires" + WHERE "cron_lease"."expires" <= now() + RETURNING "owner"`, + [job, owner, `${LEASE_TTL_SECONDS}`], + ); + + this.recordSuccess(); + + return claimed.length > 0; + } + + /** + * Pushes the expiry out while the job is still running. Returns false when this RUN is no longer + * the owner — which means the claim has lapsed and someone else has taken the job over, and this + * run should be treated as having lost it. + */ + async renew(job: string, owner: string): Promise { + const [, affected] = await this.dataSource.query( + `UPDATE "cron_lease" + SET "expires" = now() + ($3 || ' seconds')::interval + WHERE "name" = $1 AND "owner" = $2`, + [job, owner, `${LEASE_TTL_SECONDS}`], + ); + + this.recordSuccess(); + + return affected > 0; + } + + /** + * Releases the lease. Scoped to the owner that took it, so a run which already lost the lease + * cannot delete the row another run is now holding — in another process or in this one. + */ + async release(job: string, owner: string): Promise { + await this.dataSource.query(`DELETE FROM "cron_lease" WHERE "name" = $1 AND "owner" = $2`, [job, owner]); + + this.recordSuccess(); + } + + /** + * Runs `task` only if this process can claim the lease, and keeps the claim alive meanwhile. + * + * Failing to reach the database means NOT running — under `api` or `worker`. There the lease is + * the only thing keeping the job to one process, and a job that moves money must not proceed on + * the assumption that it is probably alone. The caller sees the same outcome as a job whose + * lease is held elsewhere: it does not run this cycle and tries again on the next. + * + * Under `all` it means running anyway. That role IS one process, the shape this deployment had + * before any lease existed, so stopping there would make an unreachable table strictly worse + * than its own absence — see the branch in the catch below. + * + * `reportContention` marks jobs for which losing the race is not a normal outcome. For a worker + * job it is: the other worker holds the lease and is doing the work, and the result lands in the + * database where everyone can see it. For a job whose effect is confined to the process that + * runs it, losing the race means that effect did not happen where it was needed — which is what + * `CronScope.API` describes. Nothing else can tell the two apart, so the caller says which it is. + */ + async run(job: string, task: () => Promise, reportContention = false): Promise { + // Once shutdown has begun, starting a run is worse than skipping it: `shutdown` waits on the + // jobs it found when it started, and the process exits when that wait ends. A run started + // afterwards is not in that set, so it would be cut off mid-way — before the `finally` below + // releases its lease, and, more importantly, part-way through whatever it was doing. + if (this.shuttingDown) return; + + const owner = this.newOwner(); + + let acquired: boolean; + try { + acquired = await this.acquire(job, owner); + } catch (e) { + this.recordFailure(e); + + // Unreachable table, missing grant, database down. What happens next depends on the role, + // and the difference matters more than it looks. + // + // Under `all` the deployment runs ONE process — the same shape the API had before this + // branch existed, when no lease was involved at all. Skipping there would make an + // unreachable lease table STRICTLY WORSE than not having one: 123 of 139 jobs would stop, + // payouts included, and between the rollout of this application version and the rollout of + // the alert that reads the heartbeat there is no rule that would say so. So the job runs. + // Two processes on `all` would then run it twice — exactly as they would have before, and + // the `role-mismatch` rule reports that pair once it exists. + // + // Under `api` or `worker` the lease is the only thing keeping the job to one process, and + // its absence is not recoverable by running anyway. There the skip stands, and the + // heartbeat carries the reason out. + if (Config.cronRole !== CronRole.ALL) { + this.logger.error(`Skipping ${job}: could not reach the lease table`, e); + return; + } + + this.logger.error( + `Running ${job} WITHOUT a lease: could not reach the lease table, and CRON_ROLE=all runs ` + + `one process — not running it would be worse than the single-process setup this ` + + `replaces`, + e, + ); + + // Through `track` rather than as a bare `task()`. The two things the healthy path does + // between here and the call are not about the lease at all, and skipping them would make + // this run less safe than every other: it would be invisible to `shutdown` — no grace, and + // absent from the "still running" warning — and it would start even if shutdown had begun + // during the failed claim, which is a longer window than the healthy one because the + // attempt runs to the database timeout. + return this.track(job, owner, task); + } + + if (!acquired) { + if (reportContention) + this.logger.error( + `Skipped ${job}: another process holds the lease. This job only has an effect in the ` + + `process that runs it, so that effect did not happen here`, + ); + + return; + } + + // Claiming the lease is a round trip, and shutdown can begin during it. `track` below checks + // for that as its first act and hands the claim straight back rather than starting under it, + // so the successor does not sit out the expiry for a job that never ran. + // + // This is best effort, not a guarantee: a job still inside its claim is not in `inFlight` + // yet, so `shutdown` does not wait for it, and the process can exit before the release runs. + // What then remains is a claim nobody holds — it lapses within the TTL like any other, which + // is the bound that always applies. The release only ever shortens that wait. + const renewal = this.keepAlive(job, owner); + + return this.track(job, owner, task, () => { + renewal.stop(); + + return this.release(job, owner).catch((e) => { + this.recordFailure(e); + this.logger.error(`Could not release the lease for ${job}`, e); + }); + }); + } + + /** + * Runs a task as one this process is known to be running. + * + * Everything a run needs regardless of whether it holds a claim: the shutdown check that must + * happen as late as possible, and the `inFlight` entry that makes the run visible to + * `shutdown`. Both were once written out only on the path that holds a lease, which left the + * lease-less path — the one taken when the table cannot be reached under `all` — without either. + * + * `after` is what the claim-holding path adds: stop renewing, hand the claim back. The + * lease-less path has nothing to hand back. + */ + private async track( + job: string, + owner: string, + task: () => Promise, + after?: () => Promise, + ): Promise { + // As late as possible, because the step before it is a round trip: shutdown can begin while a + // claim is being taken, or while a failing attempt runs to its timeout. A run started after + // that point is not in the set `shutdown` waits on and would be cut off part-way through. + if (this.shuttingDown) { + await after?.(); + return; + } + + const run = (async () => { + try { + await task(); + } finally { + await after?.(); + this.inFlight.delete(owner); + } + })(); + + this.inFlight.set(owner, { job, run: run.catch(() => undefined) }); + + return run; + } + + /** + * Waits for the jobs this process is still running, so their normal release path can hand the + * lease over to the successor instead of leaving it to expire. + * + * Called from a SIGTERM/SIGINT handler in `main.ts`, deliberately NOT through Nest's + * `enableShutdownHooks`. That switch is global: it would also start running nine + * `onModuleDestroy` hooks that have never run in this application, because nothing ever asked + * for a shutdown hook. Nest runs those BEFORE this one, and they empty the strategy registries + * that PayIn, PayOut and DEX jobs resolve from. Combined with the wait below — which is the + * whole point here, keeping in-flight jobs alive LONGER into the shutdown — that would let a + * running payout fail on an emptied registry instead of simply being cut off. Handing over a + * lease is not worth activating that. + * + * A lease is NOT taken away from a job that is still working. Releasing on SIGTERM would hand + * over faster, but the job keeps running until this process exits, which `main.ts` does once the + * wait below ends — so up to `SHUTDOWN_GRACE_MS` after the signal. A successor claiming the + * freed lease inside that window would run the same money-moving job alongside it, which is the + * outcome this mechanism exists to make rare, so it is not traded for a faster handover. + * + * What is still running after the wait therefore keeps its lease, which lapses within + * `LEASE_TTL_SECONDS` of the last renewal. The renewal timers deliberately keep going meanwhile: + * they hold the claim for as long as this process is alive to renew it. + * + * Bounded on every path: the only thing awaited is a race against `SHUTDOWN_GRACE_MS`, so a + * database that has stopped answering cannot turn this into a process that never exits. + */ + async shutdown(): Promise { + // Before the snapshot below, not after: the wait covers the jobs that were running when it was + // taken, and the process exits once it ends. A job that started meanwhile would not be waited + // for and would be cut off part-way through — see the guard at the top of `run`. + this.shuttingDown = true; + + const running = [...this.inFlight.values()].map((entry) => entry.run); + if (!running.length) return; + + this.logger.info(`Shutting down: waiting up to ${SHUTDOWN_GRACE_MS / 1000}s for ${running.length} running job(s)`); + + await Promise.race([Promise.all(running), this.shutdownGrace()]); + + const stranded = [...this.inFlight.values()].map((entry) => entry.job); + if (stranded.length) + this.logger.warn( + `Shutting down with ${stranded.length} job(s) still running (${stranded.join(', ')}); ` + + `their leases stay held and lapse within ${LEASE_TTL_SECONDS}s`, + ); + } + + /** + * The state of the lease layer, for the role heartbeat to report. + * + * Read rather than pushed, and it has to be read under BOTH roles because it means different + * things: under `api` or `worker` a lease that cannot reach its table stops every worker- and + * api-scoped job, and no other line says so; under `all` it stops nothing, but the jobs then run + * without a claim, which is the state to fix before the roles are split. `healthy` stays false + * until an operation succeeds, so neither case falls quiet after the first window. The counter + * is per window; the last message is not, so an unhealthy report always names something. + */ + takeFailures(): { healthy: boolean; count: number; last?: string } { + const taken = { healthy: this.healthy, count: this.failures, last: this.lastFailure }; + + this.failures = 0; + + return taken; + } + + /** + * Renews the claim for `job` while it runs, with one renewal outstanding at a time. + * + * A fixed interval fires whether or not the previous renewal has come back, and a database that + * answers slowly is exactly the situation this has to survive: the attempts pile up, each one + * occupying a pooled connection, and an older answer can land after a newer one. Re-arming only + * once the previous attempt has settled bounds that to a single outstanding statement. The price + * is that the renewals drift later by however long the database takes to answer, and the TTL — + * three times the interval — leaves room for one such answer to be slow or lost, not for a + * database that is slow to every one of them. See RENEWAL_INTERVAL_MS. + * + * Losing the claim does not stop the run. There is nothing here that could stop it, and the + * timer deliberately keeps going: this process holds the claim for as long as it can renew it. + */ + private keepAlive(job: string, owner: string): { stop: () => void } { + let stopped = false; + let timer: NodeJS.Timeout; + + const schedule = (): void => { + // Unref'd: a pending timer must never hold the process open on shutdown. + timer = setTimeout(async () => { + try { + const stillOurs = await this.renew(job, owner); + if (!stillOurs) this.logger.error(`Lost the lease for ${job} while it was still running`); + } catch (e) { + this.recordFailure(e); + this.logger.error(`Could not extend the lease for ${job}`, e); + } + + if (!stopped) schedule(); + }, RENEWAL_INTERVAL_MS); + timer.unref(); + }; + + schedule(); + + return { + stop: () => { + stopped = true; + clearTimeout(timer); + }, + }; + } + + private recordSuccess(): void { + this.healthy = true; + } + + private recordFailure(e: unknown): void { + this.failures++; + this.lastFailure = e instanceof Error ? e.message : String(e); + this.healthy = false; + } + + private shutdownGrace(): Promise { + // Unref'd so winning the race above does not keep the process alive for the rest of the grace. + return new Promise((resolve) => setTimeout(resolve, SHUTDOWN_GRACE_MS).unref()); + } +} diff --git a/src/shared/services/dfx-cron.service.ts b/src/shared/services/dfx-cron.service.ts index af3f63c068..f47be257d6 100644 --- a/src/shared/services/dfx-cron.service.ts +++ b/src/shared/services/dfx-cron.service.ts @@ -1,112 +1,279 @@ -import { Injectable, OnModuleInit } from '@nestjs/common'; -import { DiscoveryService, MetadataScanner } from '@nestjs/core'; -import { CronExpression, SchedulerRegistry } from '@nestjs/schedule'; -import { CronJob } from 'cron'; -import { Config } from 'src/config/config'; -import { DisabledProcess } from 'src/shared/services/process.service'; -import { DFX_CRONJOB_PARAMS, DfxCronExpression, DfxCronParams } from 'src/shared/utils/cron'; -import { LockClass } from 'src/shared/utils/lock'; -import { Util } from 'src/shared/utils/util'; -import { CustomCronExpression } from '../utils/custom-cron-expression'; -import { DfxLogger } from './dfx-logger'; - -interface CronJobData { - instance: object; - methodRef: any; - methodName: string; - params: DfxCronParams; -} - -@Injectable() -export class DfxCronService implements OnModuleInit { - private readonly logger = new DfxLogger(DfxCronService); - - constructor( - private readonly discovery: DiscoveryService, - private readonly metadataScanner: MetadataScanner, - private readonly schedulerRegisty: SchedulerRegistry, - ) {} - - onModuleInit() { - this.discovery - .getProviders() - .filter((wrapper) => wrapper.isDependencyTreeStatic()) - .filter(({ instance }) => instance && Object.getPrototypeOf(instance)) - .forEach(({ instance }) => { - this.metadataScanner - .getAllMethodNames(instance) - .map((methodName) => { - const methodRef = instance[methodName]; - - return { - instance, - methodRef, - methodName, - params: Reflect.getMetadata(DFX_CRONJOB_PARAMS, methodRef), - }; - }) - .filter((data) => data.params) - .forEach((data) => this.addCronJob(data)); - }); - } - - private addCronJob(data: CronJobData) { - const lock = LockClass.create(data.params.timeout ?? Infinity); - - const context = { target: data.instance.constructor.name, method: data.methodName }; - const cronJob = new CronJob(data.params.expression, () => lock(this.wrapFunction(data), context)); - const cronJobName = `${context.target}::${context.method}`; - - this.schedulerRegisty.addCronJob(cronJobName, cronJob); - cronJob.start(); - } - - private wrapFunction(data: CronJobData) { - const context = { target: data.instance.constructor.name, method: data.methodName }; - - return async (...args: any) => { - if (data.params.process && DisabledProcess(data.params.process)) { - this.logger.verbose( - `Skipping ${context.target}::${context.method} - process ${data.params.process} is disabled`, - ); - return; - } - - if (data.params.useDelay ?? true) await this.cronJobDelay(data.params.expression); - - await data.methodRef.apply(data.instance, args); - }; - } - - private async cronJobDelay(expression: DfxCronExpression): Promise { - const random = Math.random() * 1000; - - const delays = Config.cronJobDelay; - - switch (expression) { - case CronExpression.EVERY_10_SECONDS: - return Util.delay(random * (delays[0] ?? 5)); - - case CustomCronExpression.EVERY_15_SECONDS: - return Util.delay(random * (delays[1] ?? 5)); - - case CronExpression.EVERY_30_SECONDS: - return Util.delay(random * (delays[2] ?? 15)); - - case CronExpression.EVERY_MINUTE: - return Util.delay(random * (delays[3] ?? 30)); - - case CronExpression.EVERY_5_MINUTES: - return Util.delay(random * (delays[4] ?? 60)); - - case CronExpression.EVERY_10_MINUTES: - return Util.delay(random * (delays[5] ?? 60)); - - case CustomCronExpression.EVERY_15_MINUTES: - return Util.delay(random * (delays[6] ?? 60)); - - case CronExpression.EVERY_HOUR: - return Util.delay(random * (delays[7] ?? 120)); - } - } -} +import { Injectable, OnModuleInit } from '@nestjs/common'; +import { DiscoveryService, MetadataScanner } from '@nestjs/core'; +import { CronExpression, SchedulerRegistry } from '@nestjs/schedule'; +import { CronJob } from 'cron'; +import { Config, CronRole } from 'src/config/config'; +import { DisabledProcess } from 'src/shared/services/process.service'; +import { CronScope, DFX_CRONJOB_PARAMS, DfxCron, DfxCronExpression, DfxCronParams } from 'src/shared/utils/cron'; +import { LockClass } from 'src/shared/utils/lock'; +import { Util } from 'src/shared/utils/util'; +import { CustomCronExpression } from '../utils/custom-cron-expression'; +import { CronLeaseService } from './cron-lease.service'; +import { DfxLogger } from './dfx-logger'; + +interface CronJobData { + instance: object; + methodRef: any; + methodName: string; + params: DfxCronParams; +} + +@Injectable() +export class DfxCronService implements OnModuleInit { + private readonly logger = new DfxLogger(DfxCronService); + + private registeredCount = 0; + + constructor( + private readonly discovery: DiscoveryService, + private readonly metadataScanner: MetadataScanner, + private readonly schedulerRegisty: SchedulerRegistry, + private readonly leases: CronLeaseService, + ) {} + + onModuleInit() { + const registered: CronScope[] = []; + let skipped = 0; + + this.discovery + .getProviders() + .filter((wrapper) => wrapper.isDependencyTreeStatic()) + .filter(({ instance }) => instance && Object.getPrototypeOf(instance)) + .forEach(({ instance }) => { + this.metadataScanner + .getAllMethodNames(instance) + .map((methodName) => { + const methodRef = instance[methodName]; + + return { + instance, + methodRef, + methodName, + params: Reflect.getMetadata(DFX_CRONJOB_PARAMS, methodRef), + }; + }) + .filter((data) => data.params) + .forEach((data) => { + if (!this.runsInThisRole(data.params.scope)) { + skipped++; + return; + } + + registered.push(data.params.scope); + this.addCronJob(data); + }); + }); + + // Counts what this process actually registered, which is not the same as counting decorators + // in the source: a job declared on an abstract base class is registered once per concrete + // provider, and the filter above skips providers whose dependency tree is not static. Stays + // on `info` so the split is readable without changing the log level. + const total = registered.length + skipped; + const byScope = Object.values(CronScope) + .map((scope) => `${scope}: ${registered.filter((s) => s === scope).length}`) + .join(', '); + + this.registeredCount = registered.length; + + this.logger.info(`CronRole ${Config.cronRole}: registered ${registered.length} of ${total} jobs (${byScope})`); + + // And a heartbeat right away, not only at the next ten-minute mark. The line above is a + // DIFFERENT one — the alerts read the heartbeat, and a process that has just started would + // otherwise be missing from it for up to ten minutes. + // + // That is not a theoretical gap: the job fires on fixed marks (`useDelay: false`), so a + // process that comes up at :11 and misses :10 writes nothing until :20. Against a + // twelve-minute window, an ordinary deploy could then produce the "worker is silent" alarm — + // the critical one this whole split rests on. + this.reportRole(); + } + + /** + * The line above says which role this process STARTED with. It cannot answer whether the two + * processes are running the right roles right now: it is written once, so an alert built on a + * counting window over it either reports nothing after the window passes, or reports permanently. + * This line answers the same question continuously, and the alert reads it. + * + * Deliberately `both`: it has to appear in EVERY process, and it carries the role, so a swapped + * assignment shows up as a wrong role rather than only as a missing line. + * + * Deliberately without a `process` flag: a watchdog that can be switched off looks, once it is + * off, exactly like a process that stopped writing the line — the alert could not tell the two + * apart. The job holds no state and does nothing but log, so there is nothing to switch off. + * + * The line is written to be read by a machine, in exactly one of two shapes: + * + * ``` + * CronRole : heartbeat, jobs registered, lease ok + * CronRole : heartbeat, jobs registered, lease unusable: failure(s) since the last heartbeat, last error: + * ``` + * + * Three properties make that safe to match on, and all three are load-bearing. One of `lease ok` + * or `lease unusable` is ALWAYS present, so a reader sees the current state rather than having + * to count occurrences of a line that only appears when something is wrong — a count over a + * window cannot tell "healthy" from "not reporting at all". Neither literal is a prefix of the + * other. And the only free text — the reason — comes last, so every matched field sits at a + * fixed distance from the START of the line and no input can push one of them there. + * + * That last property is what a matcher has to be written to use: anchor at the start of the + * line, not at its end. The end of an unhealthy line is caller-supplied text, so a selector + * anchored there is deciding on a value the failure itself gets to choose. + * + * `__tests__/dfx-cron.service.spec.ts` pins both shapes; changing the wording here fails there. + */ + // `useDelay: false`: the alert reads this line over a 12-minute window. With the default jitter + // the gap between two heartbeats can reach 660 s, leaving 60 s of margin — and the jitter is + // configurable through CRON_JOB_DELAY, so someone could close that margin from the outside + // without ever seeing this code. A watchdog must not have its own timing tuned by a knob meant + // for spreading load. + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.BOTH, useDelay: false }) + reportRole(): void { + const line = `CronRole ${Config.cronRole}: heartbeat, ${this.registeredCount} jobs registered`; + const lease = this.leases.takeFailures(); + + // What the lease layer costs when it is broken depends on the role, and neither case says so + // by itself: under `api`/`worker` the jobs are skipped, and a skip looks exactly like a job + // with nothing to do; under `all` they run without a claim, which looks like nothing at all. + // This job is scope `both` and therefore exempt from the lease itself, so it keeps reporting + // in both cases — a count of REGISTERED jobs sees neither. + if (lease.healthy) return this.logger.info(`${line}, lease ok`); + + this.logger.error( + `${line}, lease unusable: ${lease.count} failure(s) since the last heartbeat, last error: ${ + lease.last ?? 'unknown' + }`, + ); + } + + /** + * `all` runs everything, which is the single-process mode. The other two roles each run their + * own scope plus `both`, so a job maintaining process-local state that requests read is + * registered in every process. + */ + private runsInThisRole(scope: CronScope): boolean { + switch (Config.cronRole) { + case CronRole.ALL: + return true; + + case CronRole.API: + return scope === CronScope.API || scope === CronScope.BOTH; + + case CronRole.WORKER: + return scope === CronScope.WORKER || scope === CronScope.BOTH; + } + } + + private addCronJob(data: CronJobData) { + const lock = LockClass.create(data.params.timeout ?? Infinity); + + const context = { target: data.instance.constructor.name, method: data.methodName }; + const cronJobName = `${context.target}::${context.method}`; + const run = this.guardAcrossProcesses(cronJobName, data); + const cronJob = new CronJob(data.params.expression, () => lock(run, context)); + + this.schedulerRegisty.addCronJob(cronJobName, cronJob); + cronJob.start(); + + this.logger.verbose(`Registered ${cronJobName} (${data.params.scope})`); + } + + /** + * Wraps a job in the lease, so a second process has to claim it before it can start the job. + * + * `lock` above only spans this process. Which process a job belongs to is decided by + * configuration, and configuration can be wrong — a missed recreate leaves the old role in + * place, `--scale` creates a second worker, a rollback puts two processes on `all`. In every + * one of those the two processes hold separate locks and every payout runs twice, for as long as + * it takes someone to notice. With the lease, the second process has to take the claim before it + * may start the job, and while the holder keeps renewing it never gets one. It does not rule a + * double run out, and it does not bound how long one lasts — CronLeaseService says under "What + * it does not do" exactly where it stops — so the jobs still have to tolerate a repeat. + * + * `BOTH` jobs are deliberately exempt. They exist because a request path on THIS process reads + * the state they maintain, so they have to run in every process — a lease over them would + * starve whichever process lost the race, and the job would silently stop maintaining that + * state. Their safety comes from a different property: running twice must be harmless by + * construction, which is what CONTRIBUTING requires of them. + * + * `API` jobs are leased as well, and reported when they lose the race. Their scope says their + * effect is confined to the process running them, which is an argument for every API process + * running them; the lease is what keeps one that also writes or calls out from + * routinely doing either twice. Where the two pull apart the lease wins, and the process that lost the race is left + * without whatever the job maintains — which is why an `API` job must not be the only thing + * filling what a request path reads, and why delivering to connections belongs to a `BOTH` job + * driven from stored state rather than to this scope. Losing the race is reported rather than + * passed over, because for these jobs it is the symptom of a deployment running more than one + * API process rather than a normal cycle. + */ + private guardAcrossProcesses(cronJobName: string, data: CronJobData): () => Promise { + const task = this.wrapFunction(data); + + const leased = + data.params.scope === CronScope.BOTH + ? task + : () => this.leases.run(cronJobName, task, data.params.scope === CronScope.API); + + // OUTSIDE the lease, and that order is the point: a job that is switched off must not claim + // anything. Inside, every tick of a disabled job still cost one INSERT and one DELETE on + // `cron_lease` — for the jobs that tick every second, two statements a second each, and under + // `DISABLED_PROCESSES='*'` for all of them at once. That is dead tuples on a table whose whole + // purpose is to be read quickly, produced by jobs that are doing nothing. + return this.skipWhenDisabled(leased, data); + } + + private skipWhenDisabled(task: () => Promise, data: CronJobData): () => Promise { + const { process } = data.params; + if (!process) return task; + + const context = { target: data.instance.constructor.name, method: data.methodName }; + + return async () => { + if (DisabledProcess(process)) { + this.logger.verbose(`Skipping ${context.target}::${context.method} - process ${process} is disabled`); + return; + } + + return task(); + }; + } + + private wrapFunction(data: CronJobData) { + return async (...args: any) => { + if (data.params.useDelay ?? true) await this.cronJobDelay(data.params.expression); + + await data.methodRef.apply(data.instance, args); + }; + } + + private async cronJobDelay(expression: DfxCronExpression): Promise { + const random = Math.random() * 1000; + + const delays = Config.cronJobDelay; + + switch (expression) { + case CronExpression.EVERY_10_SECONDS: + return Util.delay(random * (delays[0] ?? 5)); + + case CustomCronExpression.EVERY_15_SECONDS: + return Util.delay(random * (delays[1] ?? 5)); + + case CronExpression.EVERY_30_SECONDS: + return Util.delay(random * (delays[2] ?? 15)); + + case CronExpression.EVERY_MINUTE: + return Util.delay(random * (delays[3] ?? 30)); + + case CronExpression.EVERY_5_MINUTES: + return Util.delay(random * (delays[4] ?? 60)); + + case CronExpression.EVERY_10_MINUTES: + return Util.delay(random * (delays[5] ?? 60)); + + case CustomCronExpression.EVERY_15_MINUTES: + return Util.delay(random * (delays[6] ?? 60)); + + case CronExpression.EVERY_HOUR: + return Util.delay(random * (delays[7] ?? 120)); + } + } +} diff --git a/src/shared/services/process.service.ts b/src/shared/services/process.service.ts index 4a9f1ba2e5..c1e0ad1609 100644 --- a/src/shared/services/process.service.ts +++ b/src/shared/services/process.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { SetStaffKycClearance } from 'src/shared/auth/staff-kyc-clearance'; import { SettingService } from '../models/setting/setting.service'; -import { DfxCron } from '../utils/cron'; +import { CronScope, DfxCron } from '../utils/cron'; export enum Process { PAY_OUT = 'PayOut', @@ -118,6 +118,10 @@ export enum Process { LEDGER_MARK_TO_MARKET = 'LedgerMarkToMarket', LEDGER_CUTOVER = 'LedgerCutover', LEDGER_COA_BOOTSTRAP = 'LedgerCoaBootstrap', + DEX_PURCHASE_ORDER = 'DexPurchaseOrder', + REF_CLEANUP = 'RefCleanup', + LATEST_BALANCE_CACHE = 'LatestBalanceCache', + SPARK_TOKEN_OPTIMIZATION = 'SparkTokenOptimization', } const safetyProcesses: Process[] = [ @@ -180,7 +184,7 @@ export class ProcessService implements OnModuleInit { await this.resyncStaffKycClearance(); } - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800, scope: CronScope.BOTH }) async resyncDisabledProcesses(): Promise { const allDisabledProcesses = [ ...(await this.settingService.getDisabledProcesses()), @@ -190,20 +194,23 @@ export class ProcessService implements OnModuleInit { DisabledProcesses = this.listToMap(allDisabledProcesses); } - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800, scope: CronScope.BOTH }) async resyncDeniedJwtAddresses(): Promise { const list = await this.settingService.getDeniedJwtAddresses(); DeniedJwtAddresses = new Set(list.map((a) => a.toLowerCase())); } - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800, scope: CronScope.BOTH }) async resyncDeniedJwtAccounts(): Promise { const list = await this.settingService.getDeniedJwtAccounts(); DeniedJwtAccounts = new Set(list); } // Primes the fail-closed staff clearance allowlist — see `staff-kyc-clearance.ts` for the semantics. - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + // Both, like the sibling resync jobs above: the allowlist it fills is module-global state that a + // guard reads on the request path, so it has to be refreshed in the process serving those + // requests. Frozen at boot it fails closed, locking out staff whose clearance arrives later. + @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800, scope: CronScope.BOTH }) async resyncStaffKycClearance(): Promise { const list = await this.settingService.getStaffKycClearance(); SetStaffKycClearance(list); diff --git a/src/shared/shared.module.ts b/src/shared/shared.module.ts index 371f467022..5260a95ad1 100644 --- a/src/shared/shared.module.ts +++ b/src/shared/shared.module.ts @@ -19,6 +19,7 @@ import { CountryController } from './models/country/country.controller'; import { Country } from './models/country/country.entity'; import { CountryRepository } from './models/country/country.repository'; import { CountryService } from './models/country/country.service'; +import { CronLease } from './models/cron-lease/cron-lease.entity'; import { FiatController } from './models/fiat/fiat.controller'; import { Fiat } from './models/fiat/fiat.entity'; import { FiatRepository } from './models/fiat/fiat.repository'; @@ -35,6 +36,7 @@ import { Setting } from './models/setting/setting.entity'; import { SettingRepository } from './models/setting/setting.repository'; import { SettingService } from './models/setting/setting.service'; import { RepositoryFactory } from './repositories/repository.factory'; +import { CronLeaseService } from './services/cron-lease.service'; import { DfxCronService } from './services/dfx-cron.service'; import { HttpService } from './services/http.service'; import { PaymentInfoService } from './services/payment-info.service'; @@ -46,7 +48,10 @@ import { ProcessService } from './services/process.service'; HttpModule, ConfigModule, GeoLocationModule, - TypeOrmModule.forFeature([Asset, Fiat, Country, Language, Setting, IpLog]), + // CronLease has no repository and no service reading it through one — CronLeaseService issues + // the claim as a single statement. It is registered so `autoLoadEntities` knows the table + // belongs to the model; without that a generated migration would offer to drop it. + TypeOrmModule.forFeature([Asset, Fiat, Country, Language, Setting, IpLog, CronLease]), PassportModule.register({ defaultStrategy: 'jwt', session: true }), JwtModule.register(GetConfig().auth.jwt), I18nModule.forRoot(GetConfig().i18n), @@ -72,6 +77,7 @@ import { ProcessService } from './services/process.service'; PaymentInfoService, IpLogService, ProcessService, + CronLeaseService, DfxCronService, ], exports: [ diff --git a/src/shared/utils/async-map.ts b/src/shared/utils/async-map.ts index 445c0bdf7c..6ebea83ac3 100644 --- a/src/shared/utils/async-map.ts +++ b/src/shared/utils/async-map.ts @@ -37,6 +37,10 @@ export class AsyncMap { return Array.from(this.subscribers.keys()); } + public has(id: K): boolean { + return this.subscribers.has(id); + } + public resolve(id: K, value: T) { const subscriber = this.subscribers.get(id); if (subscriber) { diff --git a/src/shared/utils/cron.ts b/src/shared/utils/cron.ts index 7faa755747..08c76d2dd8 100644 --- a/src/shared/utils/cron.ts +++ b/src/shared/utils/cron.ts @@ -2,25 +2,70 @@ import { CronExpression } from '@nestjs/schedule'; import { Process } from '../services/process.service'; import { CustomCronExpression } from './custom-cron-expression'; +/** + * Which process a job belongs to. + * + * The distinction is not about importance but about where a job's effect is visible. Most jobs + * only touch the database or an external system, so any process can run them and exactly one + * should. The exceptions are jobs maintaining state that lives inside the process itself - a + * class field, a module-global variable, a Node process metric - because such state is only + * useful in the process whose requests read it. + */ +export enum CronScope { + /** Worker process only. The normal case: anything writing to the database or driving business forward. */ + WORKER = 'worker', + /** + * API process only. Maintains or measures state read exclusively from a request path. + * + * Not for delivering to the connections a process holds open: such a job is leased like any + * other, so it would run in one process while the connections are spread over all of them. + * Delivery is driven from stored state and scoped `BOTH` - see + * `PaymentLinkPaymentService.deliverPaymentUpdates`. + */ + API = 'api', + /** + * Every process. Maintains or measures process-local state that both sides read. + * + * Running such a job twice must be harmless by construction: refreshing an in-memory copy of + * global state, expiring a local cache or writing a log line qualifies. Writing to the + * database or driving business forward does not - cron locks are per-process and cannot + * prevent duplicate execution across processes. + */ + BOTH = 'both', +} + export interface DfxCronOptParams { process?: Process; useDelay?: boolean; timeout?: number; } +/** + * Parameters of a cron job. `scope` is mandatory and has no default. + * + * A wrong classification fails silently - a job wrongly scoped `worker` leaves the cache it + * maintains empty in the process that reads it, with no error anywhere. Requiring the field puts + * that decision in front of whoever adds a job, and the compiler enforces it. A default plus a + * list of exceptions would move the same decision into a list the compiler does not see, where a + * test can only pin the entries it already has. + */ +export interface DfxCronRequiredParams extends DfxCronOptParams { + scope: CronScope; +} + export type DfxCronExpression = CronExpression | CustomCronExpression; -export interface DfxCronParams extends DfxCronOptParams { +export interface DfxCronParams extends DfxCronRequiredParams { expression: DfxCronExpression; } export const DFX_CRONJOB_PARAMS = 'DFXCronjobParams'; -export function DfxCron(expression: DfxCronExpression, optional?: DfxCronOptParams) { - return function (target: any, propertyKey: string, descriptor: PropertyDescriptor) { +export function DfxCron(expression: DfxCronExpression, required: DfxCronRequiredParams): MethodDecorator { + return function (target: object, propertyKey: string | symbol, descriptor: PropertyDescriptor) { const methodRef = target[propertyKey]; - const params: DfxCronParams = { expression, ...optional }; + const params: DfxCronParams = { expression, ...required }; Reflect.defineMetadata(DFX_CRONJOB_PARAMS, params, methodRef); diff --git a/src/subdomains/core/accounting/services/ledger-booking-job.service.ts b/src/subdomains/core/accounting/services/ledger-booking-job.service.ts index 0675a8e3a7..6647612dee 100644 --- a/src/subdomains/core/accounting/services/ledger-booking-job.service.ts +++ b/src/subdomains/core/accounting/services/ledger-booking-job.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { BankTxConsumer } from './consumers/bank-tx.consumer'; import { BuyCryptoConsumer } from './consumers/buy-crypto.consumer'; import { BuyFiatConsumer } from './consumers/buy-fiat.consumer'; @@ -51,59 +51,95 @@ export class LedgerBookingJobService { return (await this.settingService.get(CUTOVER_LOG_ID_KEY)) != null; } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_BANK_TX, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_BANK_TX, + timeout: 1800, + }) async runBankTx(): Promise { if (!(await this.isLedgerReady())) return; await this.bankTxConsumer.process(); } // ExchangeTx + ExchangeTrade are ONE @DfxCron method → one flag (Minor R8-1): deposit/withdrawal then trade - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_EXCHANGE_TX, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_EXCHANGE_TX, + timeout: 1800, + }) async runExchangeTx(): Promise { if (!(await this.isLedgerReady())) return; await this.exchangeTxConsumer.process(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_CRYPTO_INPUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_CRYPTO_INPUT, + timeout: 1800, + }) async runCryptoInput(): Promise { if (!(await this.isLedgerReady())) return; await this.cryptoInputConsumer.process(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_PAYOUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_PAYOUT, + timeout: 1800, + }) async runPayoutOrder(): Promise { if (!(await this.isLedgerReady())) return; await this.payoutOrderConsumer.process(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_BUY_CRYPTO, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_BUY_CRYPTO, + timeout: 1800, + }) async runBuyCrypto(): Promise { if (!(await this.isLedgerReady())) return; await this.buyCryptoConsumer.process(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_BUY_FIAT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_BUY_FIAT, + timeout: 1800, + }) async runBuyFiat(): Promise { if (!(await this.isLedgerReady())) return; await this.buyFiatConsumer.process(); } // §4.8 — bridge-only (skips exchange/DfxDex movements booked by their authoritative consumers) - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_LIQUIDITY_MANAGEMENT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_LIQUIDITY_MANAGEMENT, + timeout: 1800, + }) async runLiquidityMgmt(): Promise { if (!(await this.isLedgerReady())) return; await this.liquidityMgmtConsumer.process(); } // §4.8a — DfxDex purchase/sell on-chain swaps (own flag, Hard Constraint #5) - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_LIQUIDITY_ORDER, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_LIQUIDITY_ORDER, + timeout: 1800, + }) async runLiquidityOrderDex(): Promise { if (!(await this.isLedgerReady())) return; await this.liquidityOrderDexConsumer.process(); } // §4.9 — arbitrage swaps (own flag, Hard Constraint #5) - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LEDGER_BOOKING_TRADING_ORDER, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LEDGER_BOOKING_TRADING_ORDER, + timeout: 1800, + }) async runTradingOrder(): Promise { if (!(await this.isLedgerReady())) return; await this.tradingOrderConsumer.process(); @@ -113,7 +149,11 @@ export class LedgerBookingJobService { // Frick #4252) never get an ASSET account from the cutover-only bootstrap and wedge their consumer fail-loud // ("CoA bootstrap missing"). bootstrap() is idempotent (findOrCreate, §3), so a recurring re-run is a no-op // once complete. Pre-cutover the cutover run owns the bootstrap → gate on isLedgerReady. - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.LEDGER_COA_BOOTSTRAP, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { + scope: CronScope.WORKER, + process: Process.LEDGER_COA_BOOTSTRAP, + timeout: 1800, + }) async runCoaBootstrap(): Promise { if (!(await this.isLedgerReady())) return; await this.bootstrapService.bootstrap(); diff --git a/src/subdomains/core/accounting/services/ledger-cutover.service.ts b/src/subdomains/core/accounting/services/ledger-cutover.service.ts index d803665a4b..84d3393756 100644 --- a/src/subdomains/core/accounting/services/ledger-cutover.service.ts +++ b/src/subdomains/core/accounting/services/ledger-cutover.service.ts @@ -7,7 +7,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BuyCrypto } from 'src/subdomains/core/buy-crypto/process/entities/buy-crypto.entity'; import { LiquidityManagementOrder } from 'src/subdomains/core/liquidity-management/entities/liquidity-management-order.entity'; @@ -103,7 +103,7 @@ export class LedgerCutoverService { * a crash never breaks the boot/cron run, leaves `ledgerCutoverLogId` unset → all consumers no-op (§4 gate). * The cron no-ops immediately once the flag is set, so it effectively runs once and is otherwise idle. */ - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.LEDGER_CUTOVER }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.LEDGER_CUTOVER }) async run(): Promise { // Two deliberate checks: master switch (hard off, no DB) vs. already-cut-over (setting). Do not merge them. if (!Config.ledger.enabled) return; diff --git a/src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts b/src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts index 3735660251..1333547384 100644 --- a/src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts +++ b/src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { In } from 'typeorm'; import { AccountType, LedgerAccount } from '../entities/ledger-account.entity'; @@ -45,7 +45,7 @@ export class LedgerMarkToMarketService { private readonly ledgerLegRepository: LedgerLegRepository, ) {} - @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { process: Process.LEDGER_MARK_TO_MARKET }) + @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { scope: CronScope.WORKER, process: Process.LEDGER_MARK_TO_MARKET }) async run(): Promise { if (!(await this.jobService.isLedgerReady())) return; // cutover-gate (Blocker R1-6) applies here too diff --git a/src/subdomains/core/accounting/services/ledger-reconciliation.service.ts b/src/subdomains/core/accounting/services/ledger-reconciliation.service.ts index 5bcd015f5f..f78cb3ef33 100644 --- a/src/subdomains/core/accounting/services/ledger-reconciliation.service.ts +++ b/src/subdomains/core/accounting/services/ledger-reconciliation.service.ts @@ -6,7 +6,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { LiquidityBalance } from 'src/subdomains/core/liquidity-management/entities/liquidity-balance.entity'; import { LiquidityManagementBalanceService } from 'src/subdomains/core/liquidity-management/services/liquidity-management-balance.service'; @@ -97,7 +97,7 @@ export class LedgerReconciliationService { private readonly refRewardService: RefRewardService, ) {} - @DfxCron(CronExpression.EVERY_DAY_AT_5AM, { process: Process.LEDGER_RECONCILIATION }) + @DfxCron(CronExpression.EVERY_DAY_AT_5AM, { scope: CronScope.WORKER, process: Process.LEDGER_RECONCILIATION }) async run(): Promise { if (!(await this.jobService.isLedgerReady())) return; // cutover-gate (Blocker R1-6) diff --git a/src/subdomains/core/aml/services/sanction.service.ts b/src/subdomains/core/aml/services/sanction.service.ts index 2b1a5a08d0..fbe4f21b7f 100644 --- a/src/subdomains/core/aml/services/sanction.service.ts +++ b/src/subdomains/core/aml/services/sanction.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config, Environment } from 'src/config/config'; import { HttpService } from 'src/shared/services/http.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { Sanction } from '../entities/sanction.entity'; import { SanctionRepository } from '../repositories/sanction.repository'; @@ -40,7 +40,7 @@ export class SanctionService { ) {} // --- JOBS --- // - @DfxCron(CronExpression.EVERY_WEEKEND, { process: Process.SANCTION_SYNC }) + @DfxCron(CronExpression.EVERY_WEEKEND, { scope: CronScope.WORKER, process: Process.SANCTION_SYNC }) async syncList() { const filePath = Config.environment === Environment.LOC ? this.fileName : `/home/${this.fileName}`; diff --git a/src/subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts b/src/subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts index d15e88ae8a..135229025e 100644 --- a/src/subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts +++ b/src/subdomains/core/buy-crypto/process/services/buy-crypto-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { BuyCryptoBatchService } from './buy-crypto-batch.service'; import { BuyCryptoDexService } from './buy-crypto-dex.service'; import { BuyCryptoNotificationService } from './buy-crypto-notification.service'; @@ -20,7 +20,7 @@ export class BuyCryptoJobService { private readonly buyCryptoPreparationService: BuyCryptoPreparationService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BUY_CRYPTO, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.BUY_CRYPTO, timeout: 7200 }) async process() { await this.buyCryptoRegistrationService.registerCryptoPayIn(); await this.buyCryptoRegistrationService.syncReturnTxId(); @@ -37,7 +37,11 @@ export class BuyCryptoJobService { await this.buyCryptoNotificationService.sendNotificationMails(); } - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.BUY_CRYPTO_AGGREGATION, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_HOUR, { + scope: CronScope.WORKER, + process: Process.BUY_CRYPTO_AGGREGATION, + timeout: 7200, + }) async checkAggregatingTransactions() { await this.buyCryptoPreparationService.checkAggregatingTransactions(); } diff --git a/src/subdomains/core/buy-crypto/routes/buy/buy.service.ts b/src/subdomains/core/buy-crypto/routes/buy/buy.service.ts index 3c05fd26db..c573db0713 100644 --- a/src/subdomains/core/buy-crypto/routes/buy/buy.service.ts +++ b/src/subdomains/core/buy-crypto/routes/buy/buy.service.ts @@ -15,7 +15,7 @@ import { AssetDtoMapper } from 'src/shared/models/asset/dto/asset-dto.mapper'; import { FiatDtoMapper } from 'src/shared/models/fiat/dto/fiat-dto.mapper'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { PaymentInfoService } from 'src/shared/services/payment-info.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { PdfUtil } from 'src/shared/utils/pdf.util'; import { Util } from 'src/shared/utils/util'; import { RouteService } from 'src/subdomains/core/route/route.service'; @@ -66,12 +66,12 @@ export class BuyService { ) {} // --- VOLUMES --- // - @DfxCron(CronExpression.EVERY_YEAR) + @DfxCron(CronExpression.EVERY_YEAR, { scope: CronScope.WORKER }) async resetAnnualVolumes(): Promise { await this.buyRepo.update({ annualVolume: Not(0) }, { annualVolume: 0 }); } - @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT) + @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT, { scope: CronScope.WORKER }) async resetMonthlyVolumes(): Promise { await this.buyRepo.update({ monthlyVolume: Not(0) }, { monthlyVolume: 0 }); } diff --git a/src/subdomains/core/buy-crypto/routes/swap/swap.service.ts b/src/subdomains/core/buy-crypto/routes/swap/swap.service.ts index 64657b985e..ff53d49cc5 100644 --- a/src/subdomains/core/buy-crypto/routes/swap/swap.service.ts +++ b/src/subdomains/core/buy-crypto/routes/swap/swap.service.ts @@ -17,7 +17,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { AssetService } from 'src/shared/models/asset/asset.service'; import { AssetDtoMapper } from 'src/shared/models/asset/dto/asset-dto.mapper'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BuyCryptoExtended } from 'src/subdomains/core/history/mappers/transaction-dto.mapper'; import { RouteService } from 'src/subdomains/core/route/route.service'; @@ -85,12 +85,12 @@ export class SwapService { } // --- VOLUMES --- // - @DfxCron(CronExpression.EVERY_YEAR) + @DfxCron(CronExpression.EVERY_YEAR, { scope: CronScope.WORKER }) async resetAnnualVolumes(): Promise { await this.swapRepo.update({ annualVolume: Not(0) }, { annualVolume: 0 }); } - @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT) + @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT, { scope: CronScope.WORKER }) async resetMonthlyVolumes(): Promise { await this.swapRepo.update({ monthlyVolume: Not(0) }, { monthlyVolume: 0 }); } diff --git a/src/subdomains/core/custody/services/custody-job.service.ts b/src/subdomains/core/custody/services/custody-job.service.ts index e1f3aa7d7f..2e6024b7cb 100644 --- a/src/subdomains/core/custody/services/custody-job.service.ts +++ b/src/subdomains/core/custody/services/custody-job.service.ts @@ -8,7 +8,7 @@ import { OrderConfig } from '../config/order-config'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CustodyOrderStep } from '../entities/custody-order-step.entity'; import { @@ -38,14 +38,14 @@ export class CustodyJobService { private readonly custodyOrderService: CustodyOrderService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.CUSTODY }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.CUSTODY }) async handleOrders() { await this.executeOrder(); await this.executeStep(); await this.checkStep(); } - @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { process: Process.CUSTODY }) + @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { scope: CronScope.WORKER, process: Process.CUSTODY }) async resetExpiredConfirmedOrders() { const expiryDate = Util.daysBefore(Config.txRequestWaitingExpiryDays); diff --git a/src/subdomains/core/faucet-request/services/faucet-request.service.ts b/src/subdomains/core/faucet-request/services/faucet-request.service.ts index aecf71e7f7..953bff8ad2 100644 --- a/src/subdomains/core/faucet-request/services/faucet-request.service.ts +++ b/src/subdomains/core/faucet-request/services/faucet-request.service.ts @@ -13,7 +13,7 @@ import { AssetService } from 'src/shared/models/asset/asset.service'; import { AssetDtoMapper } from 'src/shared/models/asset/dto/asset-dto.mapper'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { KycLevel } from 'src/subdomains/generic/user/models/user-data/user-data.enum'; import { UserService } from 'src/subdomains/generic/user/models/user/user.service'; import { In, Not } from 'typeorm'; @@ -36,7 +36,7 @@ export class FaucetRequestService { return [Environment.DEV, Environment.LOC].includes(Config.environment) ? Blockchain.SEPOLIA : Blockchain.ETHEREUM; } - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.CRYPTO_PAYOUT }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.CRYPTO_PAYOUT }) async checkFaucetRequests(): Promise { const pendingFaucets = await this.faucetRequestRepo.find({ where: { status: FaucetRequestStatus.IN_PROGRESS } }); for (const faucet of pendingFaucets) { diff --git a/src/subdomains/core/history/controllers/transaction.controller.ts b/src/subdomains/core/history/controllers/transaction.controller.ts index e1135c54fd..d285cf28a5 100644 --- a/src/subdomains/core/history/controllers/transaction.controller.ts +++ b/src/subdomains/core/history/controllers/transaction.controller.ts @@ -31,7 +31,7 @@ import { UserRole } from 'src/shared/auth/user-role.enum'; import { isFiatDto } from 'src/shared/models/active'; import { AssetDtoMapper } from 'src/shared/models/asset/dto/asset-dto.mapper'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { AmountType, Util } from 'src/shared/utils/util'; import { BankDataService } from 'src/subdomains/generic/user/models/bank-data/bank-data.service'; import { UserData } from 'src/subdomains/generic/user/models/user-data/user-data.entity'; @@ -116,7 +116,9 @@ export class TransactionController { ) {} // --- JOBS --- // - @DfxCron(CronExpression.EVERY_MINUTE) + // Api, not Both: `refundList` is filled and read by the refund request paths of this + // controller, so it stays empty in a process those requests never reach. + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.API }) checkLists() { for (const [key, refundData] of this.refundList.entries()) { if (!this.isRefundDataValid(refundData)) this.refundList.delete(key); diff --git a/src/subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts b/src/subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts index 8c74b457fe..da87a948ce 100644 --- a/src/subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts +++ b/src/subdomains/core/liquidity-management/services/liquidity-management-pipeline.service.ts @@ -2,7 +2,7 @@ import { BadRequestException, ConflictException, Injectable, NotFoundException } import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailRequest } from 'src/subdomains/supporting/notification/interfaces'; @@ -77,7 +77,11 @@ export class LiquidityManagementPipelineService { //*** JOBS ***// - @DfxCron(CronExpression.EVERY_10_SECONDS, { process: Process.LIQUIDITY_MANAGEMENT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_SECONDS, { + scope: CronScope.WORKER, + process: Process.LIQUIDITY_MANAGEMENT, + timeout: 1800, + }) async processPipelines(): Promise { let hasChanges = true; while (hasChanges) { diff --git a/src/subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts b/src/subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts index 0a6c68d992..eb1fc045c4 100644 --- a/src/subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts +++ b/src/subdomains/core/liquidity-management/services/liquidity-management-rule.service.ts @@ -6,7 +6,7 @@ import { Fiat } from 'src/shared/models/fiat/fiat.entity'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailRequest } from 'src/subdomains/supporting/notification/interfaces'; import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; @@ -105,7 +105,11 @@ export class LiquidityManagementRuleService { //*** JOBS ***// - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.LIQUIDITY_MANAGEMENT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { + scope: CronScope.WORKER, + process: Process.LIQUIDITY_MANAGEMENT, + timeout: 1800, + }) async reactivateRules(): Promise { const rules = await this.ruleRepo.findBy({ status: LiquidityManagementRuleStatus.PAUSED, diff --git a/src/subdomains/core/liquidity-management/services/liquidity-management.service.ts b/src/subdomains/core/liquidity-management/services/liquidity-management.service.ts index 3559b5631f..545dd86ca0 100644 --- a/src/subdomains/core/liquidity-management/services/liquidity-management.service.ts +++ b/src/subdomains/core/liquidity-management/services/liquidity-management.service.ts @@ -4,7 +4,7 @@ import { Config } from 'src/config/config'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { PriceCurrency, @@ -37,7 +37,11 @@ export class LiquidityManagementService { //*** JOBS ***// - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.LIQUIDITY_MANAGEMENT_CHECK_BALANCES, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.LIQUIDITY_MANAGEMENT_CHECK_BALANCES, + timeout: 1800, + }) async checkLiquidityBalances() { const rules = await this.ruleRepo.findBy({ status: Not(LiquidityManagementRuleStatus.DISABLED) }); const balances = await this.balanceService.refreshBalances(rules); diff --git a/src/subdomains/core/monitoring/__tests__/monitoring.service.spec.ts b/src/subdomains/core/monitoring/__tests__/monitoring.service.spec.ts new file mode 100644 index 0000000000..8459429c27 --- /dev/null +++ b/src/subdomains/core/monitoring/__tests__/monitoring.service.spec.ts @@ -0,0 +1,343 @@ +import { DeepMocked, createMock } from '@golevelup/ts-jest'; +import { NotFoundException } from '@nestjs/common'; +import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; +import { MonitoringService } from '../monitoring.service'; +import { Metric, SystemState } from '../system-state-snapshot.entity'; +import { SystemStateSnapshotRepository } from '../system-state-snapshot.repository'; + +function snapshot(state: SystemState): { id: number; data: string } { + return { id: 1, data: JSON.stringify(state) }; +} + +function metric(data: unknown, updated: string): Metric { + return { data, updated: new Date(updated) }; +} + +describe('MonitoringService', () => { + let repo: DeepMocked; + let notificationService: NotificationService; + let service: MonitoringService; + + // Timestamps in the past, so a value written during the test is always the later one. + const persisted: SystemState = { + node: { health: metric({ up: true }, '2020-01-01T00:10:00Z') }, + bank: { balance: metric({ chf: 42 }, '2020-01-01T00:10:00Z') }, + }; + + // As it comes back out of the database: JSON carries `updated` as a string, which serialises + // identically in the response. + const asRead = (): SystemState => JSON.parse(JSON.stringify(persisted)); + + let lockedReads: unknown[]; + let written: { id: number; data: string }[]; + + beforeEach(() => { + lockedReads = []; + written = []; + + repo = createMock(); + repo.findOne.mockResolvedValue(snapshot(persisted) as never); + + // Stands in for the transaction: same row content, but through a manager whose lock option + // and writes the test can inspect. + const manager = { + findOne: jest.fn().mockImplementation((_entity: unknown, options: { lock?: unknown }) => { + lockedReads.push(options?.lock); + return Promise.resolve(written.length ? written[written.length - 1] : snapshot(persisted)); + }), + save: jest.fn().mockImplementation((_entity: unknown, row: { id: number; data: string }) => { + written.push(row); + return Promise.resolve(row); + }), + }; + Object.defineProperty(repo, 'manager', { + value: { transaction: (run: (m: unknown) => Promise) => run(manager) }, + configurable: true, + }); + notificationService = createMock(); + + service = new MonitoringService(repo, notificationService); + }); + + describe('reading the state', () => { + it('answers from the persisted state, not from the in-memory state', async () => { + // Without this the endpoints behind GET /health* and /monitoring/data would answer from the + // boot snapshot in any process not running the observers. + await expect(service.getState(undefined, undefined)).resolves.toEqual(asRead()); + }); + + it('refreshes the filtered queries too', async () => { + // GET /monitoring/data takes subsystem and metric as query parameters. Refreshing only the + // unfiltered branch would leave exactly those answers stale. + await expect(service.getState('bank', undefined)).resolves.toEqual(asRead().bank); + await expect(service.getState('bank', 'balance')).resolves.toEqual(asRead().bank.balance); + }); + + it('still reports an unknown subsystem or metric as not found', async () => { + await expect(service.getState('ledger', undefined)).rejects.toThrow(NotFoundException); + await expect(service.getState('bank', 'turnover')).rejects.toThrow(NotFoundException); + }); + + it('prefers whichever value carries the later timestamp', () => { + const older: SystemState = { bank: { balance: metric({ chf: 42 }, '2020-01-01T00:10:00Z') } }; + const newer: SystemState = { bank: { balance: metric({ chf: 43 }, '2020-01-01T00:20:00Z') } }; + + expect(service['mergeNewer'](older, newer).bank.balance.data).toEqual({ chf: 43 }); + expect(service['mergeNewer'](newer, older).bank.balance.data).toEqual({ chf: 43 }); + }); + + it('shows a value produced in this process before it is persisted', async () => { + // The webhook path writes into the in-memory state of whichever process receives the call, + // and the persisted row follows only after the debounce. Reading the database alone would + // answer with the older value in that window. + await service['updateSystemState']('bank', 'balance', { chf: 99 }); + + const state = (await service.getState(undefined, undefined)) as SystemState; + + expect(state.bank.balance.data).toEqual({ chf: 99 }); + expect(state.node.health.data).toEqual({ up: true }); + }); + + it('reads at most once per cache window', async () => { + await service.getState(undefined, undefined); + await service.getState(undefined, undefined); + + expect(repo.findOne).toHaveBeenCalledTimes(1); + }); + + it('does not send a mail when the read fails', async () => { + // Unlike the load at start-up this path runs on every request: a database problem would + // otherwise answer itself with a flood of mails, precisely during the outage. + repo.findOne.mockRejectedValue(new Error('database unavailable')); + + await expect(service.getState(undefined, undefined)).resolves.toEqual({}); + expect(notificationService.sendMail).not.toHaveBeenCalled(); + }); + }); + + /** Lets the first transaction fail with the given database error, then behaves normally. */ + function failFirstWith(error: { code: string; message: string }): () => number { + let attempts = 0; + + Object.defineProperty(repo, 'manager', { + value: { + transaction: (run: (m: unknown) => Promise) => { + attempts++; + if (attempts === 1) return Promise.reject(Object.assign(new Error(error.message), { code: error.code })); + + return run({ + findOne: jest.fn().mockResolvedValue(snapshot(persisted)), + save: jest.fn().mockImplementation((_e: unknown, row: { id: number; data: string }) => { + written.push(row); + return Promise.resolve(row); + }), + }); + }, + }, + configurable: true, + }); + + return () => attempts; + } + + describe('persisting the state', () => { + it('keeps metrics another process wrote', async () => { + // Every process subscribes to its own updates and writes the same single row. Replacing it + // with this process's view would drop the other's work - the API process would overwrite + // the observers' results with its boot state. + const prev: SystemState = { bank: { balance: metric({ chf: 42 }, '2020-01-01T00:10:00Z') } }; + const next: SystemState = { bank: { balance: metric({ chf: 43 }, '2020-01-01T00:20:00Z') } }; + + await service['persist'](prev, next); + + const result = JSON.parse(written[0].data) as SystemState; + + expect(result.bank.balance.data).toEqual({ chf: 43 }); + expect(result.node.health.data).toEqual({ up: true }); + }); + + it('writes nothing when no metric changed', async () => { + await service['persist'](persisted, persisted); + + expect(written).toEqual([]); + }); + + it('reads the row under a write lock, in the same transaction it writes in', async () => { + // Without the lock, merging only narrows the race instead of closing it: two writers that + // both read before either wrote still overwrite each other - and the lost value does not + // come back on its own, because the next run compares it against this process's own + // previous state and finds it unchanged. + const prev: SystemState = { bank: { balance: metric({ chf: 42 }, '2020-01-01T00:10:00Z') } }; + const next: SystemState = { bank: { balance: metric({ chf: 43 }, '2020-01-01T00:20:00Z') } }; + + await service['persist'](prev, next); + + expect(lockedReads).toEqual([{ mode: 'pessimistic_write' }]); + expect(written).toHaveLength(1); + }); + + it('does not put an older measurement back over a newer one', async () => { + // This process may have waited on the lock while another wrote a later measurement of the + // same metric. Writing regardless would restore the older value, and it would stay until + // the metric changes here again. + const prev: SystemState = { bank: { balance: metric({ chf: 1 }, '2020-01-01T00:00:00Z') } }; + const stale: SystemState = { bank: { balance: metric({ chf: 2 }, '2020-01-01T00:05:00Z') } }; + + await service['persist'](prev, stale); + + const result = JSON.parse(written[0].data) as SystemState; + + expect(result.bank.balance.data).toEqual({ chf: 42 }); + }); + + it('retries when the row cannot be locked because it does not exist yet', async () => { + // An absent row cannot be locked, so two writers can reach the insert together and one + // loses on the primary key. Without the retry that process's change is dropped until the + // metric happens to change again. + const attempts = failFirstWith({ code: '23505', message: 'duplicate key value' }); + + const next: SystemState = { ledger: { open: metric({ count: 1 }, '2020-01-01T00:20:00Z') } }; + + await service['persist']({}, next); + + expect(attempts()).toBe(2); + expect(JSON.parse(written[0].data).ledger.open.data).toEqual({ count: 1 }); + expect(notificationService.sendMail).not.toHaveBeenCalled(); + }); + + it('does not retry an error a second attempt cannot resolve', async () => { + // The retry exists for the insert conflict and for deadlocks. Repeating a malformed row or + // a permission error would only put the same failing statement on the database twice. + const attempts = failFirstWith({ code: '42501', message: 'permission denied' }); + + await service['persist']({}, { ledger: { open: metric({ count: 1 }, '2020-01-01T00:20:00Z') } }); + + expect(attempts()).toBe(1); + expect(written).toEqual([]); + expect(notificationService.sendMail).toHaveBeenCalled(); + }); + + it('writes a metric that did not exist before', async () => { + const next: SystemState = { ledger: { open: metric({ count: 3 }, '2020-01-01T00:20:00Z') } }; + + await service['persist']({}, next); + + const result = JSON.parse(written[0].data) as SystemState; + + expect(result.ledger.open.data).toEqual({ count: 3 }); + expect(result.bank.balance.data).toEqual({ chf: 42 }); + }); + }); + + describe('an environment whose state row is not id 1', () => { + // The write path targets id 1, so the read has to prefer it — reading the highest id would let + // a second row make every read miss what is written. But whether a given database HAS that row + // is not decidable from here, and reading only id 1 would answer null forever where it does + // not: the observers are scoped to the worker, so the API process would never fill the state + // itself and the monitoring endpoints would answer 404 indefinitely. + + /** A repository holding one row under a different id. */ + function withRowAt(id: number): void { + repo.findOne.mockImplementation((options: { where?: { id?: number } }) => + Promise.resolve(options?.where?.id === 1 ? null : ({ id, data: JSON.stringify(persisted) } as never)), + ); + } + + it('answers from the row that exists instead of reporting nothing', async () => { + withRowAt(7); + + await expect(service.getState(undefined, undefined)).resolves.toEqual(asRead()); + }); + + it('still reports nothing when there is no row at all', async () => { + repo.findOne.mockResolvedValue(null as never); + + await expect(service.getState('node', 'health')).rejects.toBeInstanceOf(NotFoundException); + }); + + it('seeds id 1 with what the old row held, not only with what this process changed', async () => { + // Writing just the changed metric into a fresh id 1 would strand everything the old row + // carried — and the reads, which prefer id 1, would then answer from the partial row. + const managerFindOne = jest + .fn() + .mockImplementation((_entity: unknown, options: { where?: { id?: number } }) => + Promise.resolve(options?.where?.id === 1 ? null : { id: 7, data: JSON.stringify(persisted) }), + ); + Object.defineProperty(repo, 'manager', { + value: { + transaction: (run: (m: unknown) => Promise) => + run({ + findOne: managerFindOne, + save: jest.fn().mockImplementation((_e: unknown, row: { id: number; data: string }) => { + written.push(row); + return Promise.resolve(row); + }), + }), + }, + configurable: true, + }); + + await service['mergeIntoStoredState']([['aml', 'freeze']], { + aml: { freeze: metric({ frozen: 0 }, '2030-01-01T00:00:00Z') }, + }); + + expect(written).toHaveLength(1); + expect(written[0].id).toEqual(1); + + const saved = JSON.parse(written[0].data); + + expect(saved.aml.freeze.data).toEqual({ frozen: 0 }); + expect(saved.node.health.data).toEqual({ up: true }); + expect(saved.bank.balance.data).toEqual({ chf: 42 }); + }); + + it('merges into id 1 when another writer created it while this one waited for the lock', async () => { + // Two writers, one old row and no id 1: both miss id 1, both queue for the old row. The one + // that gets there second wakes up after the first has created id 1 and committed. Merging + // from the OLD row then writes id 1 from a state that predates it — and this path only + // writes what changed, so the first writer's metric does not come back. Asking a second + // time after the wait is what closes that. + const seededById1: SystemState = { + ...persisted, + node: { health: metric({ up: true }, '2020-01-01T00:10:00Z') }, + // What the OTHER writer put there while this one waited. + ledger: { drift: metric({ off: 3 }, '2029-01-01T00:00:00Z') }, + }; + + let idOneExists = false; + const managerFindOne = jest.fn().mockImplementation((_entity: unknown, options: { where?: { id?: number } }) => { + if (options?.where?.id === 1) { + // Missing on the first ask; present once the wait on the old row is over. + return Promise.resolve(idOneExists ? { id: 1, data: JSON.stringify(seededById1) } : null); + } + // The lock on the old row is granted only after the other writer committed. + idOneExists = true; + return Promise.resolve({ id: 7, data: JSON.stringify(persisted) }); + }); + + Object.defineProperty(repo, 'manager', { + value: { + transaction: (run: (m: unknown) => Promise) => + run({ + findOne: managerFindOne, + save: jest.fn().mockImplementation((_e: unknown, row: { id: number; data: string }) => { + written.push(row); + return Promise.resolve(row); + }), + }), + }, + configurable: true, + }); + + await service['mergeIntoStoredState']([['aml', 'freeze']], { + aml: { freeze: metric({ frozen: 0 }, '2030-01-01T00:00:00Z') }, + }); + + const saved = JSON.parse(written[0].data); + + expect(saved.aml.freeze.data).toEqual({ frozen: 0 }); + // The other writer's metric survived — that is the whole point. + expect(saved.ledger.drift.data).toEqual({ off: 3 }); + }); + }); +}); diff --git a/src/subdomains/core/monitoring/monitor-connection-pool.service.ts b/src/subdomains/core/monitoring/monitor-connection-pool.service.ts index 9ba51b4611..a5c39b614b 100644 --- a/src/subdomains/core/monitoring/monitor-connection-pool.service.ts +++ b/src/subdomains/core/monitoring/monitor-connection-pool.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { DataSource } from 'typeorm'; import { PostgresConnectionOptions } from 'typeorm/driver/postgres/PostgresConnectionOptions'; import { PostgresDriver } from 'typeorm/driver/postgres/PostgresDriver'; @@ -19,7 +19,7 @@ export class MonitorConnectionPoolService { this.dbConnectionPool = dbDriver.master; } - @DfxCron(CronExpression.EVERY_SECOND, { process: Process.MONITOR_CONNECTION_POOL }) + @DfxCron(CronExpression.EVERY_SECOND, { scope: CronScope.BOTH, process: Process.MONITOR_CONNECTION_POOL }) monitorConnectionPool() { const dbOptions = Config.database as PostgresConnectionOptions; const dbMaxPoolConnections = dbOptions.poolSize ?? 10; @@ -37,7 +37,7 @@ export class MonitorConnectionPoolService { } } - @DfxCron(CronExpression.EVERY_10_SECONDS, { process: Process.MONITOR_CONNECTION_POOL }) + @DfxCron(CronExpression.EVERY_10_SECONDS, { scope: CronScope.BOTH, process: Process.MONITOR_CONNECTION_POOL }) monitorConnectionPoolStatic() { const total = this.dbConnectionPool.totalCount; const idle = this.dbConnectionPool.idleCount; diff --git a/src/subdomains/core/monitoring/monitor-event-loop.service.ts b/src/subdomains/core/monitoring/monitor-event-loop.service.ts index 3a531a13c2..1093b7480d 100644 --- a/src/subdomains/core/monitoring/monitor-event-loop.service.ts +++ b/src/subdomains/core/monitoring/monitor-event-loop.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { monitorEventLoopDelay, performance } from 'perf_hooks'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; @Injectable() export class MonitorEventLoopService implements OnModuleDestroy { @@ -22,7 +22,7 @@ export class MonitorEventLoopService implements OnModuleDestroy { this.histogram.disable(); } - @DfxCron(CronExpression.EVERY_10_SECONDS, { process: Process.MONITOR_EVENT_LOOP }) + @DfxCron(CronExpression.EVERY_10_SECONDS, { scope: CronScope.BOTH, process: Process.MONITOR_EVENT_LOOP }) monitorEventLoop(): void { const toMs = (ns: number) => Math.round(ns / 1e6); diff --git a/src/subdomains/core/monitoring/monitoring.service.ts b/src/subdomains/core/monitoring/monitoring.service.ts index 244575ae61..e9e25a2ca1 100644 --- a/src/subdomains/core/monitoring/monitoring.service.ts +++ b/src/subdomains/core/monitoring/monitoring.service.ts @@ -1,21 +1,40 @@ import { Injectable, NotFoundException, OnModuleInit } from '@nestjs/common'; import { cloneDeep, isEqual } from 'lodash'; +import { FindOneOptions } from 'typeorm'; import { BehaviorSubject, debounceTime, pairwise } from 'rxjs'; import { DfxLogger } from 'src/shared/services/dfx-logger'; +import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; import { MetricObserver } from './metric.observer'; -import { Metric, MetricName, SubsystemName, SubsystemState, SystemState } from './system-state-snapshot.entity'; +import { + Metric, + MetricName, + SubsystemName, + SubsystemState, + SystemState, + SystemStateSnapshot, +} from './system-state-snapshot.entity'; import { SystemStateSnapshotRepository } from './system-state-snapshot.repository'; type SubsystemObservers = Map>; @Injectable() export class MonitoringService implements OnModuleInit { + private static readonly stateCacheMs = 30 * 1000; + + /** + * Postgres: unique violation, deadlock, serialization failure - all resolvable by trying again. + * Read from `code` on the error, the same way LedgerAccountService reads a unique violation. + */ + private static readonly retryableWriteCodes = ['23505', '40P01', '40001']; + private readonly logger = new DfxLogger(MonitoringService); #$state: BehaviorSubject = new BehaviorSubject({}); #observers: Map = new Map(); + #storedState?: { state: SystemState; loaded: number }; + #pendingLoad?: Promise; constructor( private systemStateSnapshotRepo: SystemStateSnapshotRepository, @@ -29,29 +48,28 @@ export class MonitoringService implements OnModuleInit { // *** PUBLIC API *** // async getState(subsystem: string, metric: string): Promise { + // Reading the in-memory state alone would only be correct in the process running the + // observers. The state is already persisted in full, so every read path takes it from there, + // with a short process cache in front. The refresh happens before the branching, not inside + // one of the branches: a filtered query is the same read and would otherwise stay stale. + const state = await this.currentState(); + if (!subsystem && !metric) { - return this.#$state.value; + return state; } if (subsystem && !metric) { - return this.getSubsystemState(subsystem); + return this.getSubsystemState(state, subsystem); } if (subsystem && metric) { - return this.getMetric(subsystem, metric); + return this.getMetric(state, subsystem, metric); } } async loadState(): Promise { try { - const latestPersistedState = await this.systemStateSnapshotRepo.findOne({ where: {}, order: { id: 'DESC' } }); - - if (!latestPersistedState) { - this.logger.warn('No monitoring state found in the database'); - return null; - } - - return JSON.parse(latestPersistedState.data); + return await this.readState(); } catch (e) { this.logger.error('Failed to parse loaded system state, defaulting to empty state:', e); @@ -120,11 +138,40 @@ export class MonitoringService implements OnModuleInit { .subscribe(([prevState, newState]) => this.persist(prevState, newState)); } + /** + * Writes only the metrics this process changed, merged into the stored state. + * + * The whole system state lives in a single row (`id: 1`), and `initState` above subscribes + * this writer in whichever process the service is instantiated in. Replacing the row with this + * instance's view would drop metrics another writer put there; merging only the changed ones + * keeps both. + * + * Read, merge and write happen inside one transaction that locks the row. Without the lock the + * merge only narrows the race instead of closing it: two writers that read before either wrote + * still overwrite each other, and the lost value does not come back on its own - the next run + * compares against this process's own previous state, finds the metric unchanged, and writes + * nothing. A value that rarely changes would stay wrong in the row until it does. + */ private async persist(prevState: SystemState, newState: SystemState) { try { - if (this.hasStateChanged(prevState, newState)) { - await this.systemStateSnapshotRepo.save({ id: 1, data: JSON.stringify(newState) }); - } + const changed = this.changedMetrics(prevState, newState); + if (!changed.length) return; + + // A second attempt covers the case where the row does not exist yet: it cannot be locked, + // so two writers can reach the insert together and one loses on the primary key. Retrying + // finds the row the other one created and merges into it, instead of dropping this + // process's change until the metric happens to change again. Restricted to the errors that + // a retry can actually resolve - a unique violation, a deadlock or a serialization failure. + // Anything else fails once and is reported. + const merged = await Util.retry( + () => this.mergeIntoStoredState(changed, newState), + 2, + 0, + undefined, + (e) => MonitoringService.retryableWriteCodes.includes((e as { code?: string })?.code), + ); + + this.#storedState = { state: merged, loaded: Date.now() }; } catch (e) { this.logger.error('Error persisting the state:', e); @@ -136,22 +183,174 @@ export class MonitoringService implements OnModuleInit { } } - private hasStateChanged(prevState: SystemState, newState: SystemState): boolean { - if (!prevState && newState) return true; + private async mergeIntoStoredState(changed: [string, string][], newState: SystemState): Promise { + return this.systemStateSnapshotRepo.manager.transaction(async (manager) => { + // Same lookup as the read path, so a row that lives under a different id is merged into + // rather than left behind: writing only what this process changed into a fresh `id: 1` + // would strand every metric the old row carried, and the reads would then prefer the new + // partial row over the complete one. + const row = await this.stateRow((options) => manager.findOne(SystemStateSnapshot, options), { + mode: 'pessimistic_write', + }); + + const stored: SystemState = row ? JSON.parse(row.data) : {}; + const state = cloneDeep(stored); + + for (const [subsystem, metric] of changed) { + const candidate = newState[subsystem][metric]; + + // The value this process holds is not automatically the newer one: it may have waited on + // the lock while another process wrote a later measurement of the same metric. Writing it + // anyway would put the older value back, and it would stay there until the metric changes + // again in this process. + if (this.updatedAt(candidate) < this.updatedAt(state[subsystem]?.[metric])) { + // Logged rather than dropped silently: the comment above says this value does not come + // back on its own, so a run of these lines is the signal that two writers are competing + // for one metric — which today should not happen, every metric has exactly one writer. + this.logger.info(`Discarding older ${subsystem}/${metric} state while merging the snapshot`); + continue; + } + + state[subsystem] = { ...(state[subsystem] ?? {}), [metric]: candidate }; + } - return Object.entries(newState).some(([subsystemName, subsystemState]) => - Object.entries(subsystemState).some(([metricName, newMetricState]) => { - const prevMetricState = prevState[subsystemName] && prevState[subsystemName][metricName]; + await manager.save(SystemStateSnapshot, { id: 1, data: JSON.stringify(state) }); - if (!prevMetricState && newMetricState) return true; + return state; + }); + } - return !isEqual(prevMetricState.data, newMetricState.data); - }), + /** The metrics whose data differs between the two states, as [subsystem, metric] pairs. */ + private changedMetrics(prevState: SystemState, newState: SystemState): [string, string][] { + return Object.entries(newState ?? {}).flatMap(([subsystemName, subsystemState]) => + Object.entries(subsystemState) + .filter( + ([metricName, newMetricState]) => + !isEqual(prevState?.[subsystemName]?.[metricName]?.data, newMetricState.data), + ) + .map(([metricName]) => [subsystemName, metricName] as [string, string]), ); } - private getSubsystemState(subsystem: string): SubsystemState { - const _subsystem = this.#$state.value[subsystem]; + /** Reads the persisted state without notifying: unlike loadState, this runs on every read. */ + private async readState(): Promise { + const row = await this.stateRow((options) => this.systemStateSnapshotRepo.findOne(options)); + + if (!row) { + this.logger.warn('No monitoring state found in the database'); + return null; + } + + return JSON.parse(row.data); + } + + /** + * The row holding the system state: `id: 1` where it exists, otherwise the highest id there is. + * + * The write path targets `id: 1`, and reading by highest id instead would let a second row make + * every read miss what is written. But whether an environment has that row is a property of its + * database, not of this code — and reading only `id: 1` would answer null forever wherever it + * does not, which since the observers are scoped to the worker means the monitoring endpoints + * would answer 404 on the API process indefinitely. + * + * So the read falls back, and the write path uses the same lookup to seed `id: 1` from what it + * finds. That converges: after the first write the row exists and the fallback stops being + * reached. Taking the highest id rather than the lowest keeps the fallback on the row the code + * before this branch wrote to. + */ + private async stateRow( + find: (options: FindOneOptions) => Promise, + lock?: FindOneOptions['lock'], + ): Promise { + const canonical = await find({ where: { id: 1 }, lock }); + if (canonical) return canonical; + + const fallback = await find({ where: {}, order: { id: 'DESC' }, lock }); + if (!fallback) return undefined; + + // Asked a second time, because the answer can have changed while this call waited. With a + // lock, the wait is on the fallback row itself: two writers both miss `id: 1`, both queue for + // the old row, and the one that gets there second wakes up in a world where the first has + // already created `id: 1` and committed. Merging from the old row then writes `id: 1` from a + // state that predates it, and the first writer's metric is gone — not stale, gone, because + // this path only writes what changed and nothing brings the rest back. + // + // Without a lock there is nothing to wait on and this is one extra read on a path that stops + // being taken as soon as the canonical row exists. + if (fallback.id !== 1) { + const converged = await find({ where: { id: 1 }, lock }); + if (converged) return converged; + } + + this.logger.warn(`No monitoring state under id 1, using id ${fallback.id} instead`); + + return fallback; + } + + /** + * The persisted state overlaid with anything this process holds more recently. Both matter: in + * a single-process setup the in-memory state is always the newer one, and a value arriving + * through the webhook is visible before the next persist. + */ + private async currentState(): Promise { + const stored = await this.storedState(); + + return stored ? this.mergeNewer(stored, this.#$state.value) : this.#$state.value; + } + + private async storedState(): Promise { + if (this.#storedState && Date.now() - this.#storedState.loaded < MonitoringService.stateCacheMs) { + return this.#storedState.state; + } + + // Concurrent requests share one read rather than each issuing their own. + if (!this.#pendingLoad) { + const load = this.readState().catch((e) => { + // No mail here, unlike loadState: this path is reached from getState, so a failing read + // would notify once per request instead of once per start-up. + this.logger.error('Failed to read the persisted system state:', e); + return null; + }); + + this.#pendingLoad = load; + void load.finally(() => { + if (this.#pendingLoad === load) this.#pendingLoad = undefined; + }); + } + + const state = await this.#pendingLoad; + if (state) this.#storedState = { state, loaded: Date.now() }; + + return state; + } + + /** Per metric, whichever of the two carries the later `updated` timestamp. */ + private mergeNewer(base: SystemState, overlay: SystemState): SystemState { + const merged = cloneDeep(base); + + for (const [subsystem, metrics] of Object.entries(overlay ?? {})) { + for (const [metric, state] of Object.entries(metrics)) { + if (this.updatedAt(state) >= this.updatedAt(merged[subsystem]?.[metric])) { + merged[subsystem] = { ...(merged[subsystem] ?? {}), [metric]: state }; + } + } + } + + return merged; + } + + /** + * Parsed JSON carries `updated` as a string, the in-memory state as a Date. A missing or + * unparsable value counts as the oldest possible, so a metric carrying one never wins a + * comparison against a readable timestamp - and never blocks one either. + */ + private updatedAt(metric?: Metric): number { + const time = metric?.updated ? new Date(metric.updated).getTime() : 0; + return Number.isFinite(time) ? time : 0; + } + + private getSubsystemState(state: SystemState, subsystem: string): SubsystemState { + const _subsystem = state[subsystem]; if (!_subsystem) { throw new NotFoundException(`Subsystem not found, name: ${subsystem}`); @@ -159,8 +358,8 @@ export class MonitoringService implements OnModuleInit { return _subsystem; } - private getMetric(subsystem: string, metric: string): Metric { - const _subsystem = this.getSubsystemState(subsystem); + private getMetric(state: SystemState, subsystem: string, metric: string): Metric { + const _subsystem = this.getSubsystemState(state, subsystem); const _metric = _subsystem[metric]; if (!_metric) { diff --git a/src/subdomains/core/monitoring/observers/aml.observer.ts b/src/subdomains/core/monitoring/observers/aml.observer.ts index b23e43ea9e..87209d5664 100644 --- a/src/subdomains/core/monitoring/observers/aml.observer.ts +++ b/src/subdomains/core/monitoring/observers/aml.observer.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; import { IsNull } from 'typeorm'; @@ -31,7 +31,7 @@ export class AmlObserver extends MetricObserver { super(monitoringService, 'payment', 'aml'); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { const data = await this.getAmlData(); diff --git a/src/subdomains/core/monitoring/observers/bank.observer.ts b/src/subdomains/core/monitoring/observers/bank.observer.ts index 622854b9ce..67a8205fb6 100644 --- a/src/subdomains/core/monitoring/observers/bank.observer.ts +++ b/src/subdomains/core/monitoring/observers/bank.observer.ts @@ -7,7 +7,7 @@ import { YapealService } from 'src/integration/bank/services/yapeal.service'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -38,7 +38,7 @@ export class BankObserver extends MetricObserver { super(monitoringService, 'bank', 'balance'); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { let data = []; diff --git a/src/subdomains/core/monitoring/observers/checkout.observer.ts b/src/subdomains/core/monitoring/observers/checkout.observer.ts index cf1f5e0f15..d8f3561451 100644 --- a/src/subdomains/core/monitoring/observers/checkout.observer.ts +++ b/src/subdomains/core/monitoring/observers/checkout.observer.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { CheckoutBalances, CheckoutService } from 'src/integration/checkout/services/checkout.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; import { CheckoutTxService } from 'src/subdomains/supporting/fiat-payin/services/checkout-tx.service'; @@ -30,7 +30,7 @@ export class CheckoutObserver extends MetricObserver { super(monitoringService, 'checkout', 'balance'); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { if (!this.checkoutService.isAvailable()) { if (!this.unavailableWarningLogged) { diff --git a/src/subdomains/core/monitoring/observers/exchange.observer.ts b/src/subdomains/core/monitoring/observers/exchange.observer.ts index 66ae0e7ff3..0506fcfb0d 100644 --- a/src/subdomains/core/monitoring/observers/exchange.observer.ts +++ b/src/subdomains/core/monitoring/observers/exchange.observer.ts @@ -5,7 +5,7 @@ import { ExchangeName } from 'src/integration/exchange/enums/exchange.enum'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -27,7 +27,7 @@ export class ExchangeObserver extends MetricObserver { super(monitoringService, 'exchange', 'volume'); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { if (DisabledProcess(Process.MONITORING)) return; diff --git a/src/subdomains/core/monitoring/observers/external-services.observer.ts b/src/subdomains/core/monitoring/observers/external-services.observer.ts index 9f36d27b65..0278e501f8 100644 --- a/src/subdomains/core/monitoring/observers/external-services.observer.ts +++ b/src/subdomains/core/monitoring/observers/external-services.observer.ts @@ -4,7 +4,7 @@ import { IbanService } from 'src/integration/bank/services/iban.service'; import { LetterService } from 'src/integration/letter/letter.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -31,7 +31,7 @@ export class ExternalServicesObserver extends MetricObserver { super(monitoringService, 'liquidity', 'trading'); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { const data = await this.getLiquidityData(); diff --git a/src/subdomains/core/monitoring/observers/node-balance.observer.ts b/src/subdomains/core/monitoring/observers/node-balance.observer.ts index 906f98fc67..6d9e7e10ec 100644 --- a/src/subdomains/core/monitoring/observers/node-balance.observer.ts +++ b/src/subdomains/core/monitoring/observers/node-balance.observer.ts @@ -4,7 +4,7 @@ import { BitcoinClient } from 'src/integration/blockchain/bitcoin/node/bitcoin-c import { BitcoinNodeType, BitcoinService } from 'src/integration/blockchain/bitcoin/services/bitcoin.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -31,7 +31,7 @@ export class NodeBalanceObserver extends MetricObserver { this.bitcoinClient = bitcoinService.getDefaultClient(BitcoinNodeType.BTC_INPUT); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch(): Promise { const data = await this.getNode(); diff --git a/src/subdomains/core/monitoring/observers/node-health.observer.ts b/src/subdomains/core/monitoring/observers/node-health.observer.ts index 78b5f37adf..f7aa07c8b3 100644 --- a/src/subdomains/core/monitoring/observers/node-health.observer.ts +++ b/src/subdomains/core/monitoring/observers/node-health.observer.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { BitcoinNodeType, BitcoinService } from 'src/integration/blockchain/bitcoin/services/bitcoin.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -45,7 +45,7 @@ export class NodeHealthObserver extends MetricObserver { this.emit(data); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.MONITORING, timeout: 360 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 360 }) async fetch(): Promise { const previousState = this.data; diff --git a/src/subdomains/core/monitoring/observers/payment.observer.ts b/src/subdomains/core/monitoring/observers/payment.observer.ts index bbcbf820d7..b3bc83a1a0 100644 --- a/src/subdomains/core/monitoring/observers/payment.observer.ts +++ b/src/subdomains/core/monitoring/observers/payment.observer.ts @@ -4,7 +4,7 @@ import { FRICK_TERMINAL_STATES } from 'src/integration/bank/dto/frick.dto'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; @@ -51,7 +51,7 @@ export class PaymentObserver extends MetricObserver { super(monitoringService, 'payment', 'combined'); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch() { const data = await this.getPayment(); diff --git a/src/subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts b/src/subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts index 8f38a447ce..5685ab4a18 100644 --- a/src/subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts +++ b/src/subdomains/core/monitoring/observers/realunit-w2w-gas.observer.ts @@ -6,7 +6,7 @@ import { SepoliaService } from 'src/integration/blockchain/sepolia/sepolia.servi import { EvmClient } from 'src/integration/blockchain/shared/evm/evm-client'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; @@ -40,7 +40,7 @@ export class RealUnitW2wGasObserver extends MetricObserver { super(monitoringService, 'realUnit', 'w2wGasBalance'); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch(): Promise { const data = await this.getData(); diff --git a/src/subdomains/core/monitoring/observers/user.observer.ts b/src/subdomains/core/monitoring/observers/user.observer.ts index 815d84d5f7..62cbfa3300 100644 --- a/src/subdomains/core/monitoring/observers/user.observer.ts +++ b/src/subdomains/core/monitoring/observers/user.observer.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MetricObserver } from 'src/subdomains/core/monitoring/metric.observer'; import { MonitoringService } from 'src/subdomains/core/monitoring/monitoring.service'; import { IsNull } from 'typeorm'; @@ -27,7 +27,7 @@ export class UserObserver extends MetricObserver { super(monitoringService, 'user', 'kyc'); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MONITORING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MONITORING, timeout: 1800 }) async fetch(): Promise { const data = await this.getUser(); diff --git a/src/subdomains/core/payment-link/controllers/__tests__/payment-link.gateway.spec.ts b/src/subdomains/core/payment-link/controllers/__tests__/payment-link.gateway.spec.ts new file mode 100644 index 0000000000..18dad62dc5 --- /dev/null +++ b/src/subdomains/core/payment-link/controllers/__tests__/payment-link.gateway.spec.ts @@ -0,0 +1,316 @@ +import { IncomingMessage } from 'http'; +import { CronScope, DFX_CRONJOB_PARAMS, DfxCronParams } from 'src/shared/utils/cron'; +import { PaymentDevice } from '../../entities/payment-link-payment.entity'; +import { PaymentLinkPaymentService } from '../../services/payment-link-payment.service'; +import { PaymentLinkGateway } from '../payment-link.gateway'; + +/** + * The gateway owns the sockets, so it is the only thing that knows which devices this process can + * deliver to. Every test here is about that ownership: an entry exists for as long as a socket + * does, and for no longer — whichever way the socket ends, and whether or not it ends politely. + */ +describe('PaymentLinkGateway', () => { + let gateway: PaymentLinkGateway; + let paymentService: jest.Mocked; + + /** A socket that records what was done to it and lets a test fire its events. */ + function socket() { + const listeners = new Map void)[]>(); + + return { + sent: [] as string[], + pings: 0, + terminated: false, + // `ws.OPEN`. A test that wants a closing socket sets it to 2 (`CLOSING`) or 3 (`CLOSED`). + readyState: 1 as 0 | 1 | 2 | 3, + send(data: string) { + this.sent.push(data); + }, + ping() { + this.pings++; + }, + terminate() { + this.terminated = true; + }, + on(event: string, listener: () => void) { + listeners.set(event, [...(listeners.get(event) ?? []), listener]); + }, + fire(event: string) { + for (const listener of listeners.get(event) ?? []) listener(); + }, + }; + } + + /** Accepts a connection the way the adapter does, through the public entry point. */ + function connect(device: string): ReturnType { + const client = socket(); + gateway.handleConnection(client, { url: `/v1/paymentLink?device=${device}` } as IncomingMessage); + + return client; + } + + const deviceIds = () => gateway.connectedDevices().map((d) => d.id); + + /** The sink the gateway registers on init; calling it is what the delivery does. */ + let deliver: (device: PaymentDevice) => boolean; + + beforeEach(() => { + paymentService = { + useDeviceSink: jest.fn().mockImplementation((sink) => (deliver = sink)), + useDeviceSource: jest.fn(), + forgetDeliveries: jest.fn(), + } as unknown as jest.Mocked; + + gateway = new PaymentLinkGateway(paymentService); + }); + + it('rejects a connection that names no device', () => { + expect(() => gateway.handleConnection(socket(), { url: '/v1/paymentLink' } as IncomingMessage)).toThrow( + 'device should not be empty', + ); + }); + + describe('what the delivery is allowed to see', () => { + it('hands the delivery a live view rather than a snapshot', () => { + // The whole point of deriving instead of mirroring: one function, asked twice, gives two + // different answers because the sockets changed underneath it. A register the gateway + // reported into could only be as current as its last report. + gateway.onModuleInit(); + + const source = (paymentService.useDeviceSource as jest.Mock).mock.calls[0][0] as () => { id: string }[]; + + expect(source()).toEqual([]); + + const client = connect('pos-1'); + expect(source().map((d) => d.id)).toEqual(['pos-1']); + + client.fire('close'); + expect(source()).toEqual([]); + }); + + it('stops reporting a device once its last connection is gone', () => { + const client = connect('pos-1'); + expect(deviceIds()).toEqual(['pos-1']); + + client.fire('close'); + + expect(deviceIds()).toEqual([]); + }); + + it('keeps reporting a device whose other connection is still open', () => { + // The failure a reference count had: one close path taken twice pushed the count below the + // number of live connections, and the device stopped being delivered to while someone was + // still listening. There is no count to push. + const first = connect('pos-1'); + const second = connect('pos-1'); + + first.fire('close'); + first.fire('close'); + + expect(deviceIds()).toEqual(['pos-1']); + + second.fire('close'); + + expect(deviceIds()).toEqual([]); + }); + + it('drops a connection that ends with an error instead of a close', () => { + // An aborted connection does not necessarily reach the close path, which is how an entry + // came to outlive its socket in the first place. + const client = connect('pos-1'); + + client.fire('error'); + + expect(deviceIds()).toEqual([]); + }); + + it('names a device once however many connections it holds, and until the last one goes', () => { + // The delivery asks for identities, not for dates: what a device is owed follows from the + // payments themselves. So a second connection adds nothing to say, and closing one of two + // takes nothing away — the device is still reachable through the other. + const first = connect('pos-1'); + connect('pos-1'); + + expect(gateway.connectedDevices()).toEqual([{ id: 'pos-1' }]); + + first.fire('close'); + expect(gateway.connectedDevices()).toEqual([{ id: 'pos-1' }]); + }); + }); + + describe('sockets that stopped answering', () => { + it('drops one that misses the ping', () => { + // Nothing else can see this happen: a peer that vanishes without closing leaves the socket + // open here and fires no event at all, so an unanswered ping is the only evidence there is. + const client = connect('pos-1'); + + gateway.checkConnections(); + + expect(client.pings).toEqual(1); + expect(deviceIds()).toEqual(['pos-1']); + + gateway.checkConnections(); + + expect(client.terminated).toBe(true); + expect(deviceIds()).toEqual([]); + }); + + it('forgets what the device was told when the sweep drops it', () => { + // The third way a command can be lost after the sink answered true, and the only one that + // produces neither a throw nor an 'error' event: the peer stopped answering without closing + // anything, so a send between the last two sweeps went into a socket nobody was reading. + // Together with the error and closing-state tests below, this closes the invariant: no path + // that can lose a command leaves its delivery marker standing. + connect('pos-1'); + + gateway.checkConnections(); + gateway.checkConnections(); + + expect(paymentService.forgetDeliveries).toHaveBeenCalledWith('pos-1'); + }); + + it('keeps one that answers the ping', () => { + // The negative side of the same check. Without it the sweep could pass by dropping every + // connection it looked at. + const client = connect('pos-1'); + + gateway.checkConnections(); + client.fire('pong'); + gateway.checkConnections(); + + expect(client.terminated).toBe(false); + expect(deviceIds()).toEqual(['pos-1']); + }); + + it('drops the entry even when terminating the socket throws', () => { + // A socket implementation that throws on terminate used to leave its entry behind, and the + // exception left both loops. The same entry then threw first on every following sweep, so + // the map it exists to bound never shrank again and no other device was ever checked. + const failing = connect('pos-1'); + failing.terminate = () => { + throw new Error('socket already destroyed'); + }; + const other = connect('pos-2'); + + gateway.checkConnections(); + + expect(() => gateway.checkConnections()).toThrow('socket already destroyed'); + expect(deviceIds()).toEqual(['pos-2']); + + // The sweep gets past it from here on, which is what makes the failure single-shot. + gateway.checkConnections(); + + expect(other.terminated).toBe(true); + expect(deviceIds()).toEqual([]); + }); + + it('runs in every process, because every process holds its own sockets', () => { + const params: DfxCronParams = Reflect.getMetadata( + DFX_CRONJOB_PARAMS, + PaymentLinkGateway.prototype.checkConnections, + ); + + expect(params.scope).toEqual(CronScope.BOTH); + }); + }); + + it('sends a command to every connection of the addressed device', () => { + gateway.onModuleInit(); + + const first = connect('pos-1'); + const second = connect('pos-1'); + const other = connect('pos-2'); + + expect(deliver({ id: 'pos-1', command: 'show-paid' })).toBe(true); + + expect(first.sent).toEqual(['show-paid']); + expect(second.sent).toEqual(['show-paid']); + expect(other.sent).toEqual([]); + }); + + it('reports nothing delivered when the device has no connection here', () => { + // `false` is what keeps the delivery from recording a command it never sent — a device + // connected to the OTHER process must stay owed. + gateway.onModuleInit(); + + expect(deliver({ id: 'pos-unknown', command: 'show-paid' })).toBe(false); + }); + + it('forgets what a device was told when its socket reports an error', () => { + // The one failure the state check cannot see: `ws` reports a send on a socket that closed + // mid-call asynchronously, through `'error'` — the sink has already answered `true` and the + // delivery has recorded the command. Letting the record age out does not repair that: it ages + // on the same cutoff the query uses, so it never outlives the payment's place in the read. + const client = connect('pos-1'); + + client.fire('error'); + + expect(paymentService.forgetDeliveries).toHaveBeenCalledWith('pos-1'); + }); + + it('keeps what a device was told when its socket closes in order', () => { + // The other direction, and the reason the record exists: an orderly close says nothing about + // a command that did go out. Forgetting here would repeat every command on every reconnect. + const client = connect('pos-1'); + + client.fire('close'); + + expect(paymentService.forgetDeliveries).not.toHaveBeenCalled(); + }); + + it('reports nothing delivered when the socket is closing, and drops it', () => { + // The path `send` cannot report: on a CLOSING or CLOSED socket `ws` takes the call quietly + // and raises the failure through `'error'` later. Without reading the state first the sink + // would answer `true`, and the delivery would record a command that never went out — against + // a state it will not send again, not even when the device reconnects. + gateway.onModuleInit(); + + const client = connect('pos-1'); + client.readyState = 2; + + expect(deliver({ id: 'pos-1', command: 'show-paid' })).toBe(false); + expect(client.sent).toEqual([]); + expect(deviceIds()).toEqual([]); + }); + + it('reports delivered when one socket is closing and another is open', () => { + gateway.onModuleInit(); + + const closing = connect('pos-1'); + closing.readyState = 3; + const open = connect('pos-1'); + + expect(deliver({ id: 'pos-1', command: 'show-paid' })).toBe(true); + expect(closing.sent).toEqual([]); + expect(open.sent).toEqual(['show-paid']); + expect(deviceIds()).toEqual(['pos-1']); + }); + + it('reports nothing delivered when every socket of the device throws, and drops them', () => { + gateway.onModuleInit(); + + const client = connect('pos-1'); + client.send = () => { + throw new Error('socket closed'); + }; + + expect(deliver({ id: 'pos-1', command: 'show-paid' })).toBe(false); + // Dropped, so the device stops being selected for delivery at all. + expect(deviceIds()).toEqual([]); + }); + + it('reports delivered when one socket takes it and another throws', () => { + gateway.onModuleInit(); + + const broken = connect('pos-1'); + broken.send = () => { + throw new Error('socket closed'); + }; + const working = connect('pos-1'); + + expect(deliver({ id: 'pos-1', command: 'show-paid' })).toBe(true); + expect(working.sent).toEqual(['show-paid']); + // The device is still reachable through the one that worked. + expect(deviceIds()).toEqual(['pos-1']); + }); +}); diff --git a/src/subdomains/core/payment-link/controllers/payment-link.gateway.ts b/src/subdomains/core/payment-link/controllers/payment-link.gateway.ts index 6ca46cb14e..ea772ecaaf 100644 --- a/src/subdomains/core/payment-link/controllers/payment-link.gateway.ts +++ b/src/subdomains/core/payment-link/controllers/payment-link.gateway.ts @@ -1,52 +1,210 @@ import { BadRequestException, OnModuleInit } from '@nestjs/common'; +import { CronExpression } from '@nestjs/schedule'; import { OnGatewayConnection, WebSocketGateway } from '@nestjs/websockets'; import { IncomingMessage } from 'http'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { PaymentDevice } from '../entities/payment-link-payment.entity'; -import { PaymentLinkPaymentService } from '../services/payment-link-payment.service'; +import { ConnectedDevice, PaymentLinkPaymentService } from '../services/payment-link-payment.service'; -type ClientMap = Map>; +/** + * The part of an accepted websocket this gateway uses. + * + * Named rather than taken wholesale so it is visible what the gateway needs from the socket, and + * so its behaviour can be exercised without standing up a server. + */ +interface PaymentSocket { + /** + * `ws`'s connection state. Read before sending, because `send` is not the place a closing + * socket reports itself: it throws only while still `CONNECTING`, and on a socket that is + * `CLOSING` or `CLOSED` it returns quietly and raises the failure through `'error'` — long + * after the caller decided the command was out. See `sendMessage`. + */ + readonly readyState: 0 | 1 | 2 | 3; + send(data: string): void; + ping(): void; + terminate(): void; + on(event: 'close' | 'error' | 'pong', listener: () => void): void; +} + +/** `ws.OPEN`. Named rather than imported, so the interface above stays the whole dependency. */ +const SOCKET_OPEN = 1; + +/** One open websocket, and what is known about it here. */ +interface Connection { + socket: PaymentSocket; + /** Cleared before each ping and set again by the pong; one missed round means it is gone. */ + responsive: boolean; +} @WebSocketGateway({ path: '/v1/paymentLink' }) export class PaymentLinkGateway implements OnGatewayConnection, OnModuleInit { - private readonly clients: ClientMap = new Map(); + private readonly clients = new Map>(); constructor(private readonly paymentService: PaymentLinkPaymentService) {} - onModuleInit() { - this.paymentService.getDeviceActivationObservable().subscribe((a) => this.sendMessage(a)); + onModuleInit(): void { + // Handed over rather than subscribed to: the delivery has to know whether the command reached + // a socket, and that is what `sendMessage` answers. A subscription could not say. + this.paymentService.useDeviceSink((device) => this.sendMessage(device)); + + // The delivery reads what is connected out of the map below instead of being told about it, so + // there is no second register to keep in step. See PaymentLinkPaymentService.connectedDevices. + this.paymentService.useDeviceSource(() => this.connectedDevices()); } - handleConnection(client: WebSocket, message: IncomingMessage) { + handleConnection(client: PaymentSocket, message: IncomingMessage): void { const device = new URLSearchParams(message.url?.split('?')[1]).get('device'); if (!device) throw new BadRequestException('device should not be empty'); this.addClient(device, client); } + /** + * The devices this process can deliver to right now, derived from the sockets it holds open. + * + * A device appears here for exactly as long as at least one of its connections is in the map. + * When it connected is deliberately not part of it: the delivery selects payments by their own + * lifetime, so a device that reconnects is owed what it was owed before — for as long as those + * payments are still inside the read (see DEVICE_DELIVERY_GRACE_SECONDS). A device that stays + * away past that gets nothing for the payments that aged out meanwhile. + */ + connectedDevices(): ConnectedDevice[] { + return [...this.clients.keys()].map((id) => ({ id })); + } + + /** + * Drops the connections that stopped answering. + * + * A peer that disappears without closing its socket — a network that went away, a device that + * went to sleep — leaves the socket open on this side, and no close or error event ever arrives. + * Nothing else in this class can tell such a socket from an idle one, so without this round trip + * it would stay in the map for as long as the process lives, and the delivery would keep + * querying for a device that is gone. + * + * Deliberately without a `process` flag: switching it off would reinstate exactly the unbounded + * growth it exists to prevent. It holds no state of its own and does nothing but drop sockets + * that failed to answer, so there is nothing a kill switch would usefully stop. + */ + @DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.BOTH }) + checkConnections(): void { + for (const [device, connections] of this.clients) { + for (const [clientId, connection] of connections) { + if (!connection.responsive) { + // Dropped from the map BEFORE the socket is terminated. The other way round, a + // `terminate` that throws left the entry in place and took the exception out of both + // loops, so every device after this one went unchecked — and the same entry threw first + // on the next sweep, and on every one after that. + this.removeClient(device, clientId); + connection.socket.terminate(); + + // And what this process believes it told that device goes with it. This is the third + // way a command can be lost after the sink answered `true`, and the only one that + // produces neither a throw nor an `'error'`: the peer stopped answering without closing + // anything, so a `send` between the last two sweeps went into a socket nobody was + // reading. An orderly `'close'` is different and deliberately does NOT do this — there + // the peer completed the closing handshake, which means it was still reading, and + // forgetting would repeat every command on every reconnect. + this.paymentService.forgetDeliveries(device); + continue; + } + + connection.responsive = false; + connection.socket.ping(); + } + } + } + // --- HELPER METHODS --- // - private addClient(device: string, client: WebSocket) { + private addClient(device: string, client: PaymentSocket): void { const clientId = Util.createUniqueId('client'); - const clients = this.clients.get(device) ?? new Map(); - clients.set(clientId, client); - this.clients.set(device, clients); + const connections = this.clients.get(device) ?? new Map(); + connections.set(clientId, { socket: client, responsive: true }); + this.clients.set(device, connections); - client.onclose = () => this.removeClient(device, clientId); + // Bound to the socket, and to every way it can end: an aborted connection reports `error`, and + // binding to `close` alone left it registered. Removal is idempotent, so both firing is fine. + client.on('close', () => this.removeClient(device, clientId)); + // `error` does one thing more than `close`: it is how `ws` reports a send on a socket that + // closed between the delivery's state check and the call itself. The delivery has already + // recorded that command as sent, so the record has to go — see + // PaymentLinkPaymentService.forgetDeliveries for why `close` must NOT do the same, and + // `checkConnections` above for the third case, which reaches neither handler. + client.on('error', () => { + this.removeClient(device, clientId); + this.paymentService.forgetDeliveries(device); + }); + client.on('pong', () => this.markResponsive(device, clientId)); } - private removeClient(device: string, clientId: string) { - const clients = this.clients.get(device); - clients?.delete(clientId); - this.clients.set(device, clients); + private removeClient(device: string, clientId: string): void { + const connections = this.clients.get(device); + if (!connections) return; + + connections.delete(clientId); + + // The device goes with its last connection. An empty map left behind would keep the device in + // `connectedDevices` above, which is the one thing that must not outlive the sockets. + if (!connections.size) this.clients.delete(device); } - private sendMessage(device: PaymentDevice) { - const clients = this.clients.get(device.id); - if (!clients) return; + private markResponsive(device: string, clientId: string): void { + const connection = this.clients.get(device)?.get(clientId); + if (connection) connection.responsive = true; + } - for (const client of clients.values()) { - client.send(device.command); + /** + * Sends to every socket of a device, drops the ones that cannot take it, and reports whether + * any of them took it. + * + * The return value is what the delivery records against: it only marks a command as delivered + * once one actually left. `false` therefore has to mean "nothing got out" — for a device with + * no connection here at all, and for one whose sockets could not take it. + * + * TWO checks, because `send` alone does not tell the difference. It throws while the socket is + * still `CONNECTING`, but a socket that is `CLOSING` or `CLOSED` takes the call without a word + * and reports the failure asynchronously through `'error'` — by which time the caller has + * already been told the command went out, and the delivery has recorded it against a state it + * will not send again. So the state is read first, and only an open socket is written to. + * + * A socket that fails either way is one the peer is no longer on. Left in the map it would keep + * the device in `connectedDevices`, so the delivery would go on selecting payments for a device + * it can no longer reach. Removal is idempotent and the `close`/`error` handlers do the same + * thing, so a socket that reports both is removed once. + * + * One failing socket does not stop the others: a device with two connections is still reachable + * through the second, and that counts as delivered. + * + * What the state check cannot cover: a socket that closes BETWEEN the check and the send. `ws` + * reports that one asynchronously through `'error'`, so this has already answered `true`. + * + * Waiting for the delivery's record to age out does NOT repair that. The record ages on the + * same cutoff the delivery's query uses, so "the record is gone" and "the payment is still in + * the read" exclude each other by construction — there is no later tick that would find both. + * The repair is in the `'error'` handler above, which drops the record so the next tick sends + * the state again. + */ + private sendMessage(device: PaymentDevice): boolean { + const connections = this.clients.get(device.id); + if (!connections) return false; + + let delivered = false; + + for (const [clientId, { socket }] of [...connections]) { + if (socket.readyState !== SOCKET_OPEN) { + this.removeClient(device.id, clientId); + continue; + } + + try { + socket.send(device.command); + delivered = true; + } catch { + this.removeClient(device.id, clientId); + } } + + return delivered; } } diff --git a/src/subdomains/core/payment-link/entities/__tests__/payment-link-payment.entity.spec.ts b/src/subdomains/core/payment-link/entities/__tests__/payment-link-payment.entity.spec.ts new file mode 100644 index 0000000000..3d21ada445 --- /dev/null +++ b/src/subdomains/core/payment-link/entities/__tests__/payment-link-payment.entity.spec.ts @@ -0,0 +1,65 @@ +import { readFileSync } from 'fs'; +import { join } from 'path'; +import { DefaultNamingStrategy, getMetadataArgsStorage } from 'typeorm'; +import { PaymentLinkPaymentMode, PaymentLinkPaymentStatus } from '../../enums'; +import { PaymentLinkPayment } from '../payment-link-payment.entity'; + +const MIGRATION = join( + __dirname, + '..', + '..', + '..', + '..', + '..', + '..', + 'migration', + '1785620000000-AddPaymentLinkPaymentDeviceIdIndex.js', +); + +describe('PaymentLinkPayment', () => { + function payment(values: Partial): PaymentLinkPayment { + return Object.assign(new PaymentLinkPayment(), { + status: PaymentLinkPaymentStatus.PENDING, + mode: PaymentLinkPaymentMode.SINGLE, + txCount: 0, + ...values, + }); + } + + describe('waitState', () => { + it('changes when the payment leaves pending', () => { + expect(payment({ status: PaymentLinkPaymentStatus.COMPLETED }).waitState).not.toEqual(payment({}).waitState); + }); + + it('changes when a MULTIPLE-mode payment counts another completed quote', () => { + const before = payment({ mode: PaymentLinkPaymentMode.MULTIPLE, txCount: 1 }); + const after = payment({ mode: PaymentLinkPaymentMode.MULTIPLE, txCount: 2 }); + + expect(after.waitState).not.toEqual(before.waitState); + }); + + it('stays the same across changes a caller is not waiting for', () => { + expect(payment({ isConfirmed: true, note: 'edited' }).waitState).toEqual(payment({}).waitState); + }); + }); + + describe('deviceId index', () => { + /** + * `deliverToConnectedDevices` looks payments up by `deviceId`, and the index backing that is + * created by a hand-written migration. Nothing in the running application compares the two — + * the next `npm run migration` does, and an entity that has drifted from the migration produces + * a migration "fixing" the difference in the direction of the entity. + */ + it('is declared on the entity under the name the migration creates', () => { + const declared = getMetadataArgsStorage().indices.filter((index) => index.target === PaymentLinkPayment); + + expect(declared.map((index) => index.columns)).toContainEqual(['deviceId']); + + const name = new DefaultNamingStrategy().indexName('payment_link_payment', ['deviceId']); + + expect(readFileSync(MIGRATION, 'utf8')).toContain( + `CREATE INDEX "${name}" ON "payment_link_payment" ("deviceId")`, + ); + }); + }); +}); diff --git a/src/subdomains/core/payment-link/entities/payment-link-payment.entity.ts b/src/subdomains/core/payment-link/entities/payment-link-payment.entity.ts index c16fe33ebb..ef434cbe0a 100644 --- a/src/subdomains/core/payment-link/entities/payment-link-payment.entity.ts +++ b/src/subdomains/core/payment-link/entities/payment-link-payment.entity.ts @@ -49,6 +49,7 @@ export class PaymentLinkPayment extends IEntity { @Column({ default: false }) isConfirmed: boolean; + @Index() @Column({ length: 256, nullable: true }) deviceId?: string; @@ -99,4 +100,18 @@ export class PaymentLinkPayment extends IEntity { get device(): PaymentDevice | undefined { return this.deviceId && this.deviceCommand ? { id: this.deviceId, command: this.deviceCommand } : undefined; } + + /** + * The persisted state a caller of `PaymentLinkPaymentService.waitForPayment` is released on. + * + * `status` alone is not enough: a `MULTIPLE`-mode payment stays `Pending` while its quotes + * complete one after another, and each of those releases the callers waiting at that moment + * (see `PaymentLinkPaymentService.handleQuoteChange`). What changes there is `txCount`. + * + * It is therefore a value to compare against the one the payment carried when the wait started, + * not a predicate: for a `MULTIPLE`-mode payment there is no absolute "released" state to test. + */ + get waitState(): string { + return `${this.status}:${this.txCount}`; + } } diff --git a/src/subdomains/core/payment-link/services/__tests__/payment-cron.service.spec.ts b/src/subdomains/core/payment-link/services/__tests__/payment-cron.service.spec.ts new file mode 100644 index 0000000000..782fcbcde7 --- /dev/null +++ b/src/subdomains/core/payment-link/services/__tests__/payment-cron.service.spec.ts @@ -0,0 +1,27 @@ +import { CronScope, DFX_CRONJOB_PARAMS, DfxCronParams } from 'src/shared/utils/cron'; +import { PaymentCronService } from '../payment-cron.service'; + +/** + * The scopes of these three jobs are the whole point of the split between writing and delivering, + * and nothing at runtime notices when one of them changes: a writing job scoped `both` would + * duplicate its webhooks, and the delivery job scoped `worker` or `api` would take the lease and + * leave the callers of every other process unreleased. Both are silent. + */ +describe('PaymentCronService', () => { + function scopeOf(method: keyof PaymentCronService): CronScope { + const params: DfxCronParams = Reflect.getMetadata(DFX_CRONJOB_PARAMS, PaymentCronService.prototype[method]); + + return params?.scope; + } + + it.each(['processExpiredPayments', 'checkTxConfirmations', 'forwardDeposits'] as const)( + 'runs %s in one process, because it writes and calls out', + (method) => { + expect(scopeOf(method)).toEqual(CronScope.WORKER); + }, + ); + + it('runs the delivery in every process, because each holds its own callers and devices', () => { + expect(scopeOf('deliverPaymentUpdates')).toEqual(CronScope.BOTH); + }); +}); diff --git a/src/subdomains/core/payment-link/services/__tests__/payment-link-fee.service.spec.ts b/src/subdomains/core/payment-link/services/__tests__/payment-link-fee.service.spec.ts index 9f2634deb8..5d2ae2ccb3 100644 --- a/src/subdomains/core/payment-link/services/__tests__/payment-link-fee.service.spec.ts +++ b/src/subdomains/core/payment-link/services/__tests__/payment-link-fee.service.spec.ts @@ -1,9 +1,29 @@ +import { ConfigService, CronRole, Environment, GetConfig } from 'src/config/config'; import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; import { BlockchainRegistryService } from 'src/integration/blockchain/shared/services/blockchain-registry.service'; import { PayoutBitcoinService } from 'src/subdomains/supporting/payout/services/payout-bitcoin.service'; import { PayoutFiroService } from 'src/subdomains/supporting/payout/services/payout-firo.service'; import { PaymentLinkFeeService } from '../payment-link-fee.service'; +/** Rebuilds the global `Config` from the current environment, the way the config spec does. */ +function withEnv(vars: Record): () => void { + const previous = Object.keys(vars).map((key) => [key, process.env[key]] as const); + + for (const [key, value] of Object.entries(vars)) { + if (value == null) delete process.env[key]; + else process.env[key] = value; + } + new ConfigService(GetConfig()); + + return () => { + for (const [key, value] of previous) { + if (value == null) delete process.env[key]; + else process.env[key] = value; + } + new ConfigService(GetConfig()); + }; +} + describe('PaymentLinkFeeService', () => { let service: PaymentLinkFeeService; let blockchainRegistryService: jest.Mocked; @@ -63,4 +83,90 @@ describe('PaymentLinkFeeService', () => { expect(fee).toBe(1); }); }); + + // --- getMinFee() Tests --- // + + describe('getMinFee()', () => { + let restore: () => void; + + afterEach(() => restore?.()); + + it('should load on demand when the cache is empty', async () => { + restore = withEnv({ ENVIRONMENT: Environment.DEV }); + + await expect(service.getMinFee(Blockchain.BITCOIN)).resolves.toBe(4); + expect(payoutBitcoinService.getRecommendedFeeRate).toHaveBeenCalledTimes(1); + }); + + it('should serve the second call from the cache the first one filled', async () => { + restore = withEnv({ ENVIRONMENT: Environment.DEV }); + + await service.getMinFee(Blockchain.BITCOIN); + await expect(service.getMinFee(Blockchain.BITCOIN)).resolves.toBe(4); + + expect(payoutBitcoinService.getRecommendedFeeRate).toHaveBeenCalledTimes(1); + }); + + it('should start ONE load for concurrent callers of the same blockchain', async () => { + restore = withEnv({ ENVIRONMENT: Environment.DEV }); + + // The cache is written when the load resolves, so without the in-flight map each of these + // would see the same empty entry and start its own call. + const fees = await Promise.all([ + service.getMinFee(Blockchain.BITCOIN), + service.getMinFee(Blockchain.BITCOIN), + service.getMinFee(Blockchain.BITCOIN), + ]); + + expect(fees).toEqual([4, 4, 4]); + expect(payoutBitcoinService.getRecommendedFeeRate).toHaveBeenCalledTimes(1); + }); + + it('should let the next caller retry after a failed load instead of inheriting it', async () => { + restore = withEnv({ ENVIRONMENT: Environment.DEV }); + payoutBitcoinService.getRecommendedFeeRate.mockRejectedValueOnce(new Error('node down')); + + await expect(service.getMinFee(Blockchain.BITCOIN)).resolves.toBeUndefined(); + await expect(service.getMinFee(Blockchain.BITCOIN)).resolves.toBe(4); + + expect(payoutBitcoinService.getRecommendedFeeRate).toHaveBeenCalledTimes(2); + }); + + it('should not reach out to any fee source on LOC', async () => { + // There are no node connections locally, so the job returns early and never fills the cache. + // Loading on demand here would run into a timeout on every request. + restore = withEnv({ ENVIRONMENT: Environment.LOC }); + + await expect(service.getMinFee(Blockchain.BITCOIN)).resolves.toBeUndefined(); + expect(payoutBitcoinService.getRecommendedFeeRate).not.toHaveBeenCalled(); + }); + }); + + // --- onModuleInit() Tests --- // + + describe('onModuleInit()', () => { + let restore: () => void; + + afterEach(() => restore?.()); + + it('should NOT warm the cache in the worker process', () => { + // The hook runs in every process regardless of CRON_ROLE; the scope on `updateFees` does not + // reach it. Unguarded it would query every fee source to fill a map nothing there reads. + restore = withEnv({ CRON_ROLE: CronRole.WORKER }); + const updateFees = jest.spyOn(service, 'updateFees').mockResolvedValue(); + + service.onModuleInit(); + + expect(updateFees).not.toHaveBeenCalled(); + }); + + it.each([CronRole.ALL, CronRole.API])('should warm the cache where requests land (%s)', (role) => { + restore = withEnv({ CRON_ROLE: role }); + const updateFees = jest.spyOn(service, 'updateFees').mockResolvedValue(); + + service.onModuleInit(); + + expect(updateFees).toHaveBeenCalledTimes(1); + }); + }); }); diff --git a/src/subdomains/core/payment-link/services/__tests__/payment-link-payment.service.spec.ts b/src/subdomains/core/payment-link/services/__tests__/payment-link-payment.service.spec.ts new file mode 100644 index 0000000000..2d460c23e9 --- /dev/null +++ b/src/subdomains/core/payment-link/services/__tests__/payment-link-payment.service.spec.ts @@ -0,0 +1,847 @@ +import { ConfigService, GetConfig } from 'src/config/config'; +import { BlockchainRegistryService } from 'src/integration/blockchain/shared/services/blockchain-registry.service'; +import { FiatService } from 'src/shared/models/fiat/fiat.service'; +import { Util } from 'src/shared/utils/util'; +import { EntityManager, In } from 'typeorm'; +import { PaymentDevice, PaymentLinkPayment } from '../../entities/payment-link-payment.entity'; +import { PaymentQuote } from '../../entities/payment-quote.entity'; +import { PaymentLinkPaymentMode, PaymentLinkPaymentStatus, PaymentQuoteStatus } from '../../enums'; +import { PaymentLinkPaymentRepository } from '../../repositories/payment-link-payment.repository'; +import { PaymentActivationService } from '../payment-activation.service'; +import { PaymentLinkPaymentService } from '../payment-link-payment.service'; +import { PaymentQuoteService } from '../payment-quote.service'; +import { PaymentWebhookService } from '../payment-webhook.service'; + +/** + * The delivery channels of this service are process-local, while the jobs writing the payments run + * in one process only. Every test here therefore describes one process: what it holds, what it may + * read, and what it delivers — never who wrote the row it reads. + */ +describe('PaymentLinkPaymentService', () => { + let service: PaymentLinkPaymentService; + let paymentLinkPaymentRepo: jest.Mocked; + let paymentWebhookService: jest.Mocked; + let paymentQuoteService: jest.Mocked; + let paymentActivationService: jest.Mocked; + + /** + * Times are relative to now because the delivery read is: it asks for payments whose own end has + * not passed by more than the grace, so a payment dated at a fixed calendar point would sit + * outside every read these tests set up. + * + * `expiryDate` is what the read selects on — deliberately a column no later write moves. The + * default puts a payment inside the read; a test that wants one outside says so. + */ + function payment(values: Partial): PaymentLinkPayment { + return Object.assign(new PaymentLinkPayment(), { + id: 7, + status: PaymentLinkPaymentStatus.PENDING, + mode: PaymentLinkPaymentMode.SINGLE, + txCount: 0, + updated: Util.secondsBefore(1), + expiryDate: Util.minutesAfter(5), + link: {}, + ...values, + }); + } + + /** Resolves to the payment if it was delivered, and to a marker if nothing was. */ + async function delivery(waiting: Promise): Promise { + return Promise.race([ + waiting, + new Promise((resolve) => setTimeout(() => resolve('nothing delivered'), 50)), + ]); + } + + /** + * Stands in for the gateway's sockets. Records what was handed over AND answers whether it got + * out — the delivery only marks a command as delivered on `true`, so a sink that always said + * yes would hide exactly the failure the record has to survive. + */ + function devices(delivers = true): PaymentDevice[] { + const seen: PaymentDevice[] = []; + service.useDeviceSink((device) => { + seen.push(device); + return delivers; + }); + + return seen; + } + + /** + * Stands in for the gateway's socket map. The service reads the connected devices out of it on + * every delivery, so a device is connected here for exactly as long as this map says so — there + * is no register in the service to register with. + */ + let sockets: Set; + + function connect(deviceId: string): void { + sockets.add(deviceId); + } + + /** The rows the database holds for the delivery read below. */ + let rows: PaymentLinkPayment[]; + + /** + * Answers the delivery read out of `rows`, honouring EVERY condition each clause carries — the + * devices asked for, the cutoff, and the state that makes a payment worth delivering. Which rows + * the read admits is the whole subject of the tests that use this, so a mock that returned a + * fixed list whatever it was asked for would prove nothing about them. + */ + function findByCutoff(options: unknown): PaymentLinkPayment[] { + const clauses = ( + options as { + where: { + deviceId: { value: string[] }; + expiryDate: { value: Date }; + status?: { value: PaymentLinkPaymentStatus }; + txCount?: { value: number }; + }[]; + } + ).where; + + return rows.filter((row) => + clauses.some( + (clause) => + clause.deviceId.value.includes(row.deviceId) && + row.expiryDate > clause.expiryDate.value && + (clause.status == null || row.status !== clause.status.value) && + (clause.txCount == null || row.txCount > clause.txCount.value), + ), + ); + } + + /** + * The row the transition competes for, and the only thing that says whether a caller won: the + * manager below applies an update exactly when its criteria still match, as the database does. + */ + let row: PaymentLinkPayment; + + /** + * Stands in for the transaction the transition runs in, statements included. Statements are + * staged and written back to `row` only when the callback returns — a callback that throws + * leaves the row as it was, which is what a rollback means and what these tests are about. + */ + function transaction(run: (manager: EntityManager) => Promise): Promise { + const staged = Object.assign(new PaymentLinkPayment(), row); + + return run(transactionManager(staged)).then((result) => { + Object.assign(row, staged); + + return result; + }); + } + + function transactionManager(staged: PaymentLinkPayment): EntityManager { + const update = jest.fn().mockImplementation((_target, criteria: number | Partial, values) => { + const matches = typeof criteria === 'number' || criteria.status == null || criteria.status === staged.status; + if (matches) Object.assign(staged, values); + + return Promise.resolve({ affected: matches ? 1 : 0 }); + }); + + managerUpdates.push(update); + + const manager = { update } as unknown as EntityManager; + managers.push(manager); + + return manager; + } + + /** + * The managers handed to the transitions, in order. An effect reached through anything else is + * a statement of its own: it commits whether the transition does or not, which is the half-state + * the transaction exists to rule out — and nothing about the effect itself shows which it was. + */ + let managers: EntityManager[]; + + /** Every statement any transition ran, in order, as [criteria, values]. */ + function transitions(): [Partial, Partial][] { + return managerUpdates.flatMap((update) => + update.mock.calls.map(([, criteria, values]) => [criteria, values] as [never, never]), + ); + } + + let managerUpdates: jest.Mock[]; + + beforeEach(() => { + // The delivery reads `Config.payment.timeoutDelay` on every tick, deliberately: the span it + // reaches back over follows the configured delay rather than a copy taken at construction. + new ConfigService(GetConfig()); + + row = payment({ id: 7 }); + rows = []; + managerUpdates = []; + managers = []; + + paymentLinkPaymentRepo = { + find: jest.fn().mockResolvedValue([]), + save: jest.fn().mockImplementation((entity) => entity), + manager: { transaction: jest.fn().mockImplementation(transaction) }, + } as unknown as jest.Mocked; + + paymentWebhookService = { sendWebhook: jest.fn() } as unknown as jest.Mocked; + + paymentQuoteService = { cancelAllForPayment: jest.fn() } as unknown as jest.Mocked; + + paymentActivationService = { closeAllForPayment: jest.fn() } as unknown as jest.Mocked; + + service = new PaymentLinkPaymentService( + {} as unknown as jest.Mocked, + paymentLinkPaymentRepo, + paymentWebhookService, + paymentQuoteService, + paymentActivationService, + {} as unknown as jest.Mocked, + ); + + sockets = new Set(); + service.useDeviceSource(() => [...sockets].map((id) => ({ id }))); + }); + + afterEach(() => { + // Set by the test that raises it; left behind it would silently widen every later read. + delete process.env.PAYMENT_TIMEOUT_DELAY; + }); + + // --- deliverPaymentUpdates() Tests --- // + + describe('deliverPaymentUpdates()', () => { + it('should release a caller waiting here on a payment another process wrote', async () => { + const waiting = service.waitForPayment(payment({ id: 7 })); + + // Nothing in this process wrote the payment, so nothing in this process released the caller. + expect(await delivery(waiting)).toEqual('nothing delivered'); + + paymentLinkPaymentRepo.find.mockResolvedValue([payment({ id: 7, status: PaymentLinkPaymentStatus.COMPLETED })]); + await service.deliverPaymentUpdates(); + + expect(await delivery(waiting)).toMatchObject({ id: 7, status: PaymentLinkPaymentStatus.COMPLETED }); + }); + + it('should release a caller on a MULTIPLE-mode payment that stays pending', async () => { + const waiting = service.waitForPayment(payment({ id: 7, mode: PaymentLinkPaymentMode.MULTIPLE, txCount: 1 })); + + paymentLinkPaymentRepo.find.mockResolvedValue([ + payment({ id: 7, mode: PaymentLinkPaymentMode.MULTIPLE, txCount: 2 }), + ]); + await service.deliverPaymentUpdates(); + + expect(await delivery(waiting)).toMatchObject({ id: 7, txCount: 2 }); + }); + + it('should keep waiting while the payment is unchanged', async () => { + const waiting = service.waitForPayment(payment({ id: 7 })); + + paymentLinkPaymentRepo.find.mockResolvedValue([payment({ id: 7 })]); + await service.deliverPaymentUpdates(); + + expect(await delivery(waiting)).toEqual('nothing delivered'); + }); + + it('should not touch the database while this process holds neither a caller nor a device', async () => { + await service.deliverPaymentUpdates(); + + expect(paymentLinkPaymentRepo.find).not.toHaveBeenCalled(); + }); + + it('should ask only for the payments this process is waiting on', async () => { + void service.waitForPayment(payment({ id: 7 })); + + await service.deliverPaymentUpdates(); + + expect(paymentLinkPaymentRepo.find).toHaveBeenCalledTimes(1); + expect(paymentLinkPaymentRepo.find).toHaveBeenCalledWith(expect.objectContaining({ where: { id: In([7]) } })); + }); + + it('should send the command to a device connected here after another process wrote the payment', async () => { + const seen = devices(); + connect('pos-1'); + + paymentLinkPaymentRepo.find.mockResolvedValue([ + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + }), + ]); + await service.deliverPaymentUpdates(); + + expect(seen).toEqual([{ id: 'pos-1', command: 'show-paid' }]); + }); + + it('should send the same payment state to a device once', async () => { + // The window admits the same payment on every tick it spans, so what keeps the command from + // being repeated is the record of what was sent, not the read. + const seen = devices(); + connect('pos-1'); + + paymentLinkPaymentRepo.find.mockResolvedValue([ + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + }), + ]); + await service.deliverPaymentUpdates(); + await service.deliverPaymentUpdates(); + + expect(seen).toHaveLength(1); + }); + + it('should send both payments of a device that carry the same state', async () => { + // One slot per device could hold only the later of the two, and the next tick would then find + // the earlier one undelivered again — the two would take turns evicting each other for as + // long as the window spans both. + const seen = devices(); + connect('pos-1'); + + const paid = (id: number) => + payment({ id, status: PaymentLinkPaymentStatus.COMPLETED, deviceId: 'pos-1', deviceCommand: 'show-paid' }); + + paymentLinkPaymentRepo.find.mockResolvedValue([paid(7), paid(8)]); + await service.deliverPaymentUpdates(); + await service.deliverPaymentUpdates(); + + expect(seen).toHaveLength(2); + }); + + it('should stop looking for a device the moment the gateway no longer holds it', async () => { + // Nothing tells the service the device went away, and nothing has to: it reads the connected + // devices on every delivery, so a device that is gone simply stops appearing. + connect('pos-1'); + + await service.deliverPaymentUpdates(); + expect(paymentLinkPaymentRepo.find).toHaveBeenCalledTimes(1); + + sockets.delete('pos-1'); + + await service.deliverPaymentUpdates(); + expect(paymentLinkPaymentRepo.find).toHaveBeenCalledTimes(1); + }); + + it('should ask for every connected device in one read, on one cutoff', async () => { + // The cutoff is a property of the payments, not of the connections, so there is nothing left + // for a per-device clause to express — and one clause per device is what made the read grow + // with the number of connections. + connect('pos-1'); + connect('pos-2'); + + await service.deliverPaymentUpdates(); + + const { where } = paymentLinkPaymentRepo.find.mock.calls[0][0]; + const clauses = where as { deviceId: { value: string[] }; expiryDate: { value: Date } }[]; + + expect(clauses).toHaveLength(2); + for (const clause of clauses) { + expect(clause.deviceId.value).toEqual(['pos-1', 'pos-2']); + expect(clause.expiryDate.value).toEqual(clauses[0].expiryDate.value); + } + }); + + it('should reach back past the delay before an expiry is even acted on', async () => { + // processExpiredPayments expires a payment at its expiryDate PLUS this delay, so a cutoff + // measured from the expiryDate alone would drop the payment out of the read before the + // transition it is waiting for has happened. Reading the configured value rather than + // assuming it is what keeps that true when the value changes. + process.env.PAYMENT_TIMEOUT_DELAY = '3600'; + new ConfigService(GetConfig()); + connect('pos-1'); + + await service.deliverPaymentUpdates(); + + const { where } = paymentLinkPaymentRepo.find.mock.calls[0][0]; + const [clause] = where as { expiryDate: { value: Date } }[]; + + expect(clause.expiryDate.value.getTime()).toBeLessThan(Util.minutesBefore(60).getTime()); + }); + + it('should still deliver a payment whose write was outlived by the read beside it', async () => { + // The failure this replaces: a read bounded by `updated` asks for a span before the present, + // and a transaction that stays open longer than that span commits a row whose stamp is + // already past the far end — it is never read again. Selecting on a column no later write + // moves cannot do that: a late commit makes the row appear later, never skip. + const seen = devices(); + connect('pos-1'); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + rows = [ + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + // Stamped by a transaction that then took an hour to commit. + updated: Util.minutesBefore(60), + expiryDate: Util.minutesAfter(5), + }), + ]; + await service.deliverPaymentUpdates(); + + expect(seen).toEqual([{ id: 'pos-1', command: 'show-paid' }]); + }); + + it('should owe a reconnecting device exactly what it was owed before', async () => { + // A record tied to the connection is lost with it, and the read then starts at the new + // connection time: what completed just before the reconnect falls between the two and is + // never delivered, while everything before it is delivered again. + const seen = devices(); + connect('pos-1'); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + const paid = (id: number) => + payment({ + id, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + }); + + rows = [paid(7)]; + await service.deliverPaymentUpdates(); + expect(seen).toHaveLength(1); + + sockets.delete('pos-1'); + // Completes while nothing is connected for it. + rows = [paid(7), paid(8)]; + await service.deliverPaymentUpdates(); + + connect('pos-1'); + await service.deliverPaymentUpdates(); + + // The one it missed, and only that one: the payment of the first tick is not repeated. + expect(seen).toHaveLength(2); + }); + + it('should still deliver a payment that becomes visible after one stamped alike', async () => { + // A payment carries the stamp its write gave it and appears only once that write commits, so + // a read running beside it sees the stamp of a row it cannot see yet. A mark advanced to the + // newest stamp read would ask for something strictly newer on the next tick and never see + // that row at all. + const seen = devices(); + connect('pos-1'); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + const updated = Util.secondsBefore(1); + const paid = (id: number) => + payment({ + id, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + updated, + }); + + rows = [paid(7)]; + await service.deliverPaymentUpdates(); + expect(seen).toHaveLength(1); + + // The slower write commits. Its row was stamped at the same moment as the one already read. + rows = [paid(7), paid(8)]; + await service.deliverPaymentUpdates(); + + // Delivered — and the payment of the first tick was not sent a second time. + expect(seen).toHaveLength(2); + }); + + it('should keep owing a command whose send did not get out', async () => { + // The loss this closes: the record is keyed by DEVICE and outlives the connection, so a + // command marked delivered before a failing send would never be retried — not even when + // the device reconnects, because the record still says it heard this state. Recording only + // what got out is what keeps the periodic delivery able to repair it. + const seen = devices(false); + connect('pos-1'); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + rows = [ + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + }), + ]; + + await service.deliverPaymentUpdates(); + expect(seen).toHaveLength(1); + // Nothing got out, so nothing is recorded. + expect(service['deviceDeliveries'].get('pos-1')?.size ?? 0).toEqual(0); + + // The next tick tries again — which is the whole point. + await service.deliverPaymentUpdates(); + expect(seen).toHaveLength(2); + }); + + it('should stop repeating once a command does get out', async () => { + // The other direction: a sink that takes it must end the repetition, or the fix above would + // have traded a silent loss for an endless one. + const seen = devices(); + connect('pos-1'); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + rows = [ + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + }), + ]; + + await service.deliverPaymentUpdates(); + await service.deliverPaymentUpdates(); + + expect(seen).toHaveLength(1); + }); + + it('should forget a payment the read has left behind, and the device with its last one', async () => { + // What bounds the record is the same cutoff the query uses: a payment the read can no longer + // return cannot be delivered again, so nothing is kept for it. Without that, a device + // connected all day would accumulate an entry per payment it ever saw — and a device that + // never comes back would keep a map of its own for good. + connect('pos-1'); + // A sink that takes it: the record is only written for a command that got out. + devices(); + paymentLinkPaymentRepo.find.mockImplementation(async (options) => findByCutoff(options)); + + // Delivered directly by the writing process, which does not consult the cutoff. + rows = []; + service['deliverToDevice']( + payment({ + id: 7, + status: PaymentLinkPaymentStatus.COMPLETED, + deviceId: 'pos-1', + deviceCommand: 'show-paid', + // Its own end is long past, and past the grace that follows it. + expiryDate: Util.hoursBefore(2), + }), + ); + expect(service['deviceDeliveries'].get('pos-1').size).toEqual(1); + + await service.deliverPaymentUpdates(); + + expect(service['deviceDeliveries'].has('pos-1')).toBe(false); + }); + + it('should agree with the read at the exact boundary', () => { + // Record and read must decide "still owed?" identically, including AT the cutoff: the read + // uses MoreThan, so a payment sitting exactly on it is no longer returned — and its entry + // has to go with it. An entry one millisecond inside stays. If the two comparators ever + // drift apart, one side re-sends what the other considers settled. + const cutoff = new Date(); + service['deviceDeliveries'].set( + 'pos-1', + new Map([ + [1, { state: 'x', expiryDate: cutoff }], + [2, { state: 'y', expiryDate: new Date(cutoff.getTime() + 1) }], + ]), + ); + + service['pruneDeliveries'](cutoff); + + expect([...service['deviceDeliveries'].get('pos-1').keys()]).toEqual([2]); + }); + }); + + // --- waitForPayment() Tests --- // + + describe('waitForPayment()', () => { + it('should answer with the payment on hand once the wait elapses', async () => { + // The endpoints keep their shape: a caller that is still there is told what the payment looks + // like now, which for a pending one means "not yet, ask again". + jest.useFakeTimers({ doNotFake: ['setImmediate', 'nextTick'] }); + + try { + const pending = payment({ id: 7 }); + const waiting = service.waitForPayment(pending); + + jest.advanceTimersByTime(60_000); + + await expect(waiting).resolves.toBe(pending); + } finally { + jest.useRealTimers(); + } + }); + + it('should leave nothing behind when the wait elapses', async () => { + // The reason the wait is bounded at all. A client that hangs up says nothing the server can + // hear, so an unbounded wait left both entries in place for the lifetime of the process. + jest.useFakeTimers({ doNotFake: ['setImmediate', 'nextTick'] }); + + try { + const waiting = service.waitForPayment(payment({ id: 7 })); + + jest.advanceTimersByTime(60_000); + await waiting; + + expect(service['waitStates'].size).toEqual(0); + expect(service['paymentWaitMap'].get()).toEqual([]); + } finally { + jest.useRealTimers(); + } + }); + + it('should keep the compared state while a wait on the same payment is still registered', async () => { + // Two callers share one entry, so the one leaving must not take the state the other is + // comparing against with it. + const first = service.waitForPayment(payment({ id: 7 })); + void service.waitForPayment(payment({ id: 7 })); + + paymentLinkPaymentRepo.find.mockResolvedValue([payment({ id: 7, status: PaymentLinkPaymentStatus.COMPLETED })]); + await service.deliverPaymentUpdates(); + + await expect(first).resolves.toMatchObject({ status: PaymentLinkPaymentStatus.COMPLETED }); + }); + }); + + // --- doSave() Tests --- // + + describe('doSave()', () => { + it('should release a caller in the writing process without waiting for the job', async () => { + const pending = payment({ id: 7, deviceId: 'pos-1', deviceCommand: 'show-paid' }); + const seen = devices(); + connect('pos-1'); + + const waiting = service.waitForPayment(pending); + await service.expirePayment(pending); + + expect(await delivery(waiting)).toMatchObject({ id: 7, status: PaymentLinkPaymentStatus.EXPIRED }); + expect(seen).toEqual([{ id: 'pos-1', command: 'show-paid' }]); + }); + + it('should not repeat a delivery the writing process already made', async () => { + const pending = payment({ id: 7, deviceId: 'pos-1', deviceCommand: 'show-paid' }); + const seen = devices(); + connect('pos-1'); + + await service.expirePayment(pending); + + paymentLinkPaymentRepo.find.mockResolvedValue([pending]); + await service.deliverPaymentUpdates(); + + expect(seen).toHaveLength(1); + }); + }); + + // --- Leaving Pending --- // + + /** + * Leaving `Pending` has to be decided by the database, not by a status read a moment earlier. + * The expiry job is a `Worker` job, the expiry timers stay in the process that served the + * request, and cancelling and completing arrive from request paths — so several processes can + * hold the same payment as `Pending` at the same time, and each of them would send the merchant + * its webhook and cancel the quotes again. + * + * A row that no longer reads `Pending` is what a caller that lost meets, and it is the whole + * assertion: nothing after the transition may happen for it. + */ + describe('leaving Pending', () => { + it('should expire through a conditional update rather than a status read', async () => { + await service.expirePayment(payment({ id: 7 })); + + expect(transitions()).toEqual([ + [{ id: 7, status: PaymentLinkPaymentStatus.PENDING }, { status: PaymentLinkPaymentStatus.EXPIRED }], + ]); + }); + + it('should run the effects of an expiry on the manager of its transition', async () => { + // The effects are what the transition carries with it. Reached through the repository's own + // manager instead, they would be statements outside it: committed while the status update + // rolls back, or committed after it and lost when the caller stops in between. + await service.expirePayment(payment({ id: 7 })); + + expect(managers).toHaveLength(1); + expect(paymentQuoteService.cancelAllForPayment).toHaveBeenCalledWith(7, managers[0]); + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledWith(7, managers[0]); + }); + + it('should not expire a second time when another process took the transition', async () => { + row.status = PaymentLinkPaymentStatus.EXPIRED; + + await service.expirePayment(payment({ id: 7, link: { webhookUrl: 'https://merchant.example/hook' } as never })); + + expect(paymentWebhookService.sendWebhook).not.toHaveBeenCalled(); + expect(paymentQuoteService.cancelAllForPayment).not.toHaveBeenCalled(); + expect(paymentActivationService.closeAllForPayment).not.toHaveBeenCalled(); + expect(paymentLinkPaymentRepo.save).not.toHaveBeenCalled(); + }); + + it('should cancel through the same transition', async () => { + await service.cancelByPayment(payment({ id: 7 })); + + expect(transitions()).toEqual([ + [{ id: 7, status: PaymentLinkPaymentStatus.PENDING }, { status: PaymentLinkPaymentStatus.CANCELLED }], + ]); + expect(managers).toHaveLength(1); + expect(paymentQuoteService.cancelAllForPayment).toHaveBeenCalledWith(7, managers[0]); + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledWith(7, managers[0]); + }); + + it('should not cancel a payment the worker expired in between', async () => { + row.status = PaymentLinkPaymentStatus.EXPIRED; + + await service.cancelByPayment(payment({ id: 7, link: { webhookUrl: 'https://merchant.example/hook' } as never })); + + expect(paymentWebhookService.sendWebhook).not.toHaveBeenCalled(); + expect(paymentQuoteService.cancelAllForPayment).not.toHaveBeenCalled(); + expect(paymentLinkPaymentRepo.save).not.toHaveBeenCalled(); + }); + + /** + * What a transition costs when it commits on its own: the row leaves `Pending` and the effects + * that belong to it do not follow. `processExpiredPayments` asks for `Pending`, so such a row + * is out of reach of every job — it is not a delayed repair but a permanent half-state. + */ + describe('a caller that stops between the transition and its effects', () => { + /** The payments the expiry job would find on its next run. */ + function pendingPayments(): PaymentLinkPayment[] { + return row.status === PaymentLinkPaymentStatus.PENDING ? [row] : []; + } + + beforeEach(() => { + // The real job runs here: what makes the half-state permanent is its own query. + new ConfigService(GetConfig()); + paymentLinkPaymentRepo.find.mockImplementation(async () => pendingPayments()); + }); + + it('should ask for nothing but Pending, which is why a half-state is out of its reach', async () => { + await service.processExpiredPayments(); + + expect(paymentLinkPaymentRepo.find).toHaveBeenCalledWith( + expect.objectContaining({ + where: expect.objectContaining({ status: PaymentLinkPaymentStatus.PENDING }), + }), + ); + }); + + it('should leave the payment where the next run of the job picks it up', async () => { + paymentQuoteService.cancelAllForPayment.mockRejectedValue(new Error('connection reset')); + + await expect(service.processExpiredPayments()).rejects.toThrow('connection reset'); + + // Not expired, and its quotes are not cancelled either: both or neither. + expect(row.status).toEqual(PaymentLinkPaymentStatus.PENDING); + expect(paymentActivationService.closeAllForPayment).not.toHaveBeenCalled(); + + // And the next run finds it, which is the property the whole transition rests on. + paymentQuoteService.cancelAllForPayment.mockResolvedValue(undefined); + await service.processExpiredPayments(); + + expect(row.status).toEqual(PaymentLinkPaymentStatus.EXPIRED); + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledTimes(1); + }); + + it('should have cancelled the quotes before anything a merchant can hold up', async () => { + // The webhook is the one effect that stays outside the transaction, so it must be the last + // one: a payment whose merchant call fails is finished in the database all the same. + paymentLinkPaymentRepo.save.mockRejectedValue(new Error('merchant unreachable')); + + await expect(service.processExpiredPayments()).rejects.toThrow('merchant unreachable'); + + expect(row.status).toEqual(PaymentLinkPaymentStatus.EXPIRED); + expect(paymentQuoteService.cancelAllForPayment).toHaveBeenCalledTimes(1); + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledTimes(1); + + // Nothing left over: the job does not see it again, and does not have to. + expect(pendingPayments()).toEqual([]); + }); + }); + + /** The third way out of `Pending`, reached from a request path and from checkTxConfirmations. */ + describe('completing on a quote', () => { + function completing(values: Partial = {}): PaymentLinkPayment { + return payment({ + id: 7, + link: { configObj: { minCompletionStatus: PaymentQuoteStatus.TX_MEMPOOL } } as never, + ...values, + }); + } + + const quote = { id: 3, status: PaymentQuoteStatus.TX_MEMPOOL } as PaymentQuote; + + beforeEach(() => { + paymentQuoteService.getCompletedQuoteCount = jest.fn().mockResolvedValue(1); + paymentActivationService.closeAllForQuote = jest.fn(); + }); + + it('should complete a SINGLE payment through the transition', async () => { + await service['handleQuoteChange'](completing(), quote); + + expect(transitions()).toEqual([ + [{ id: 7, status: PaymentLinkPaymentStatus.PENDING }, { status: PaymentLinkPaymentStatus.COMPLETED }], + [7, { txCount: 1 }], + ]); + // Both statements on the one manager the transition was given: a count written through a + // second transaction would commit on its own, which is what carrying it here rules out. + expect(managers).toHaveLength(1); + expect(managers[0].update).toHaveBeenCalledTimes(2); + expect(paymentLinkPaymentRepo.save).toHaveBeenCalledTimes(1); + }); + + it('should close the activations inside the transition, not before it', async () => { + // The half-state nothing can repair: activations closed while the payment is still + // `Pending`. The quote is already final, so checkTxConfirmations does not come back to + // it, and processExpiredPayments only ever asks for `Pending` — it would expire a payment + // whose activations have been closed for good. Running the close on the transition's own + // manager is what ties the two together. + await service['handleQuoteChange'](completing(), quote); + + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledWith(7, managers[0]); + }); + + it('should leave the activations open when the transition is lost', async () => { + // Another process took the payment out of `Pending` first. Then this caller performs no + // effect at all — closing activations for a transition it did not win would be the same + // half-state seen from the other side. + row.status = PaymentLinkPaymentStatus.COMPLETED; + + await service['handleQuoteChange'](completing(), quote); + + expect(paymentActivationService.closeAllForPayment).not.toHaveBeenCalled(); + expect(paymentLinkPaymentRepo.save).not.toHaveBeenCalled(); + }); + + it('should still close the activations on the paths that take no transition', async () => { + // A payment that is no longer `Pending` has nothing to transition, but its final quote's + // activations still have to be closed — that path predates the transaction and stays. + const payment = completing(); + payment.status = PaymentLinkPaymentStatus.EXPIRED; + + await service['handleQuoteChange'](payment, quote); + + expect(paymentActivationService.closeAllForPayment).toHaveBeenCalledWith(7, undefined); + expect(transitions()).toEqual([]); + }); + + it('should carry the counted quotes into the transition, not only into the save after it', async () => { + // A completed payment is looked at by no job, so a count left behind by a caller that + // stopped after the transition would stay wrong for good. + paymentLinkPaymentRepo.save.mockRejectedValue(new Error('merchant unreachable')); + + await expect(service['handleQuoteChange'](completing(), quote)).rejects.toThrow('merchant unreachable'); + + expect(row.status).toEqual(PaymentLinkPaymentStatus.COMPLETED); + expect(row.txCount).toEqual(1); + }); + + it('should not complete it again when another process got there first', async () => { + row.status = PaymentLinkPaymentStatus.COMPLETED; + + await service['handleQuoteChange'](completing(), quote); + + expect(paymentLinkPaymentRepo.save).not.toHaveBeenCalled(); + }); + + it('should count a MULTIPLE payment without taking a transition', async () => { + // It stays `Pending`, so there is nothing to claim — and claiming would keep every process + // but one from recording the quote it counted. + await service['handleQuoteChange'](completing({ mode: PaymentLinkPaymentMode.MULTIPLE }), quote); + + expect(transitions()).toEqual([]); + expect(paymentLinkPaymentRepo.save).toHaveBeenCalledTimes(1); + }); + }); + }); +}); diff --git a/src/subdomains/core/payment-link/services/__tests__/transactional-effects.spec.ts b/src/subdomains/core/payment-link/services/__tests__/transactional-effects.spec.ts new file mode 100644 index 0000000000..e84224ba37 --- /dev/null +++ b/src/subdomains/core/payment-link/services/__tests__/transactional-effects.spec.ts @@ -0,0 +1,108 @@ +import { EntityManager } from 'typeorm'; +import { PaymentActivation } from '../../entities/payment-activation.entity'; +import { PaymentActivationService } from '../payment-activation.service'; +import { PaymentQuoteService } from '../payment-quote.service'; + +/** + * The other half of the transaction proof. + * + * `payment-link-payment.service.spec.ts` pins that the transitions HAND their effect services the + * manager they were given. That alone proves nothing about where the statement lands: a service + * that accepts the argument and then writes through its own repository would pass that test and + * still commit on its own — which is exactly the half-state the transaction exists to rule out, + * and the review that asked for this test was right that nothing here ruled it out. + * + * So this side asserts the opposite direction: given a manager, the effect goes through THAT + * manager's repository and never through the injected one. Given none, it goes through the + * injected one — the paths outside a transition still work. + */ +describe('effects that run inside a caller transaction', () => { + type RepoMock = { update: jest.Mock; find: jest.Mock; save: jest.Mock }; + + let injectedRepo: RepoMock; + let managerRepo: RepoMock; + let manager: EntityManager; + + /** The write surfaces both effect services use; `find` answers with one row so `save` runs. */ + function repoMock(): RepoMock { + return { + update: jest.fn().mockResolvedValue({ affected: 1 }), + find: jest.fn().mockResolvedValue([{ cancel: () => ({ id: 1 }) }]), + save: jest.fn().mockResolvedValue(undefined), + }; + } + + /** Every way a repository is written through, so a test cannot miss one by naming the wrong. */ + function writes(repo: RepoMock): number { + return repo.update.mock.calls.length + repo.save.mock.calls.length; + } + + beforeEach(() => { + injectedRepo = repoMock(); + managerRepo = repoMock(); + manager = { getRepository: jest.fn().mockReturnValue(managerRepo) } as unknown as EntityManager; + }); + + describe('PaymentActivationService.closeAllForPayment', () => { + function service(): PaymentActivationService { + // Only the repository and the one collaborator the constructor calls are real; the rest is + // untouched by the path under test. + const lightningService = { getDefaultClient: () => undefined } as never; + + const u = undefined as never; + + return new PaymentActivationService(lightningService, injectedRepo as unknown as never, u, u, u, u, u); + } + + it('writes through the manager it was given, not through its own repository', async () => { + await service().closeAllForPayment(7, manager); + + expect(manager.getRepository).toHaveBeenCalledWith(PaymentActivation); + expect(writes(managerRepo)).toEqual(1); + // The one that would commit on its own. + expect(writes(injectedRepo)).toEqual(0); + }); + + it('falls back to its own repository when there is no transaction to join', async () => { + await service().closeAllForPayment(7); + + expect(writes(injectedRepo)).toEqual(1); + expect(writes(managerRepo)).toEqual(0); + }); + + it('closes exactly the activations of that payment that are still open', async () => { + // The criteria matter as much as the connection: closing an already closed activation is + // harmless, closing another payment's is not. + await service().closeAllForPayment(7, manager); + + const [criteria] = managerRepo.update.mock.calls[0]; + + expect(criteria.payment).toEqual({ id: 7 }); + expect(criteria.status).toBeDefined(); + }); + }); + + describe('PaymentQuoteService.cancelAllForPayment', () => { + function service(): PaymentQuoteService { + const u = undefined as never; + + return new PaymentQuoteService(injectedRepo as unknown as never, u, u, u, u, u, u, u, u, u); + } + + it('writes through the manager it was given, not through its own repository', async () => { + await service().cancelAllForPayment(7, manager); + + expect(managerRepo.find).toHaveBeenCalledTimes(1); + expect(writes(managerRepo)).toEqual(1); + expect(injectedRepo.find).not.toHaveBeenCalled(); + expect(writes(injectedRepo)).toEqual(0); + }); + + it('falls back to its own repository when there is no transaction to join', async () => { + await service().cancelAllForPayment(7); + + expect(writes(injectedRepo)).toEqual(1); + expect(writes(managerRepo)).toEqual(0); + }); + }); +}); diff --git a/src/subdomains/core/payment-link/services/payment-activation.service.ts b/src/subdomains/core/payment-link/services/payment-activation.service.ts index dd5bd9b967..4a98268874 100644 --- a/src/subdomains/core/payment-link/services/payment-activation.service.ts +++ b/src/subdomains/core/payment-link/services/payment-activation.service.ts @@ -11,7 +11,7 @@ import { AssetService } from 'src/shared/models/asset/asset.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Util } from 'src/shared/utils/util'; import { C2BPaymentLinkService } from 'src/subdomains/core/payment-link/services/c2b-payment-link.service'; -import { Equal, LessThan, Not } from 'typeorm'; +import { EntityManager, Equal, LessThan, Not, Repository } from 'typeorm'; import { TransferInfo } from '../dto/payment-link.dto'; import { PaymentActivation } from '../entities/payment-activation.entity'; import { PaymentLinkPayment } from '../entities/payment-link-payment.entity'; @@ -46,8 +46,11 @@ export class PaymentActivationService { ); } - async closeAllForPayment(paymentId: number): Promise { - await this.paymentActivationRepo.update( + /** `manager` runs the closes in the caller's transaction, see `cancelAllForPayment` for why. */ + async closeAllForPayment(paymentId: number, manager?: EntityManager): Promise { + const repo: Repository = manager?.getRepository(PaymentActivation) ?? this.paymentActivationRepo; + + await repo.update( { payment: { id: paymentId }, status: Not(PaymentActivationStatus.CLOSED) }, { status: PaymentActivationStatus.CLOSED }, ); diff --git a/src/subdomains/core/payment-link/services/payment-cron.service.ts b/src/subdomains/core/payment-link/services/payment-cron.service.ts index 08d9fdb93c..54d78bb05c 100644 --- a/src/subdomains/core/payment-link/services/payment-cron.service.ts +++ b/src/subdomains/core/payment-link/services/payment-cron.service.ts @@ -1,35 +1,76 @@ -import { Injectable } from '@nestjs/common'; -import { CronExpression } from '@nestjs/schedule'; -import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; -import { PaymentActivationService } from './payment-activation.service'; -import { PaymentBalanceService } from './payment-balance.service'; -import { PaymentLinkPaymentService } from './payment-link-payment.service'; -import { PaymentQuoteService } from './payment-quote.service'; - -@Injectable() -export class PaymentCronService { - constructor( - private readonly paymentLinkPaymentService: PaymentLinkPaymentService, - private readonly paymentActivationService: PaymentActivationService, - private readonly paymentQuoteService: PaymentQuoteService, - private readonly paymentBalanceService: PaymentBalanceService, - ) {} - - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAYMENT_EXPIRATION }) - async processExpiredPayments(): Promise { - await this.paymentLinkPaymentService.processExpiredPayments(); - await this.paymentActivationService.processExpiredActivations(); - await this.paymentQuoteService.processExpiredQuotes(); - } - - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAYMENT_CONFIRMATIONS }) - async checkTxConfirmations(): Promise { - await this.paymentLinkPaymentService.checkTxConfirmations(); - } - - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.PAYMENT_FORWARDING }) - async forwardDeposits(): Promise { - await this.paymentBalanceService.forwardDeposits(); - } -} +import { Injectable } from '@nestjs/common'; +import { CronExpression } from '@nestjs/schedule'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; +import { CustomCronExpression } from 'src/shared/utils/custom-cron-expression'; +import { PaymentActivationService } from './payment-activation.service'; +import { PaymentBalanceService } from './payment-balance.service'; +import { PaymentLinkPaymentService } from './payment-link-payment.service'; +import { PaymentQuoteService } from './payment-quote.service'; + +@Injectable() +export class PaymentCronService { + constructor( + private readonly paymentLinkPaymentService: PaymentLinkPaymentService, + private readonly paymentActivationService: PaymentActivationService, + private readonly paymentQuoteService: PaymentQuoteService, + private readonly paymentBalanceService: PaymentBalanceService, + ) {} + + // The three jobs below split what used to be one decision. Writing and delivering have opposite + // requirements — a database write, a merchant webhook and a quote cancellation must happen once + // in the deployment, while the AsyncMap and the device sink in PaymentLinkPaymentService are + // process-local and only reach a caller connected to the process that fires them. A single scope + // cannot satisfy both: `Worker` or `Api` leaves callers on every other process unreleased, + // `Both` repeats every write and every webhook. + // + // So the writing runs under the lease (`Worker`), and deliverPaymentUpdates delivers from the + // persisted state those writes leave behind, in every process, without a lease. It writes + // nothing and calls nothing outside its process, which is what allows it to run everywhere. + + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAYMENT_EXPIRATION }) + async processExpiredPayments(): Promise { + await this.paymentLinkPaymentService.processExpiredPayments(); + await this.paymentActivationService.processExpiredActivations(); + await this.paymentQuoteService.processExpiredQuotes(); + } + + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAYMENT_CONFIRMATIONS }) + async checkTxConfirmations(): Promise { + await this.paymentLinkPaymentService.checkTxConfirmations(); + } + + // Runs at 15 seconds rather than the minute the two jobs above run at, because it is the second + // hop of a chain: the writing job already costs up to a minute to notice, and this must not add + // another one to it. It stays cheap at that rate by looking only at what its own process holds — + // with no caller waiting and no device connected it issues no query at all. + // + // `useDelay: false` for the same reason: the jitter exists to spread jobs that do real work per + // run, and up to five seconds of it would be a third of this interval. + // + // Deliberately WITHOUT a `process` flag, and that is a correction. It carried one, and the flag + // looked like any other job's — but this job is not work, it is the bridge that carries a result + // from the process that wrote it to the process holding the connection. Switched off in the + // single-process setup nothing happens, because `doSave` delivers directly there; switched off + // after the split it silently cuts delivery to everything attached to the OTHER container: + // waiting callers of `GET /v1/paymentLink/payment/wait` and `GET /v1/lnurlp/wait/:id` hang until + // they give up, and connected devices are never told their payment went through. No alert sees + // it — every process still reports its role and a usable lease. + // + // A switch whose failure mode is invisible is worse than no switch. The same reasoning already + // applies to the role heartbeat and to `PaymentLinkGateway.checkConnections`: a mechanism the + // rest depends on does not get a kill switch. Whatever a switch here would have been used for — + // load, a misbehaving device — is reached by disabling the jobs that WRITE, which do have flags. + @DfxCron(CustomCronExpression.EVERY_15_SECONDS, { + scope: CronScope.BOTH, + useDelay: false, + }) + async deliverPaymentUpdates(): Promise { + await this.paymentLinkPaymentService.deliverPaymentUpdates(); + } + + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.PAYMENT_FORWARDING }) + async forwardDeposits(): Promise { + await this.paymentBalanceService.forwardDeposits(); + } +} diff --git a/src/subdomains/core/payment-link/services/payment-link-fee.service.ts b/src/subdomains/core/payment-link/services/payment-link-fee.service.ts index ef2c786d66..309d0f757b 100644 --- a/src/subdomains/core/payment-link/services/payment-link-fee.service.ts +++ b/src/subdomains/core/payment-link/services/payment-link-fee.service.ts @@ -1,115 +1,215 @@ -import { Injectable, OnModuleInit } from '@nestjs/common'; -import { CronExpression } from '@nestjs/schedule'; -import { Environment, GetConfig } from 'src/config/config'; -import { MIN_FEE_RATE_SAT_VB } from 'src/integration/blockchain/bitcoin/services/bitcoin-based-fee.service'; -import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; -import { PaymentLinkBlockchains } from 'src/integration/blockchain/shared/util/blockchain.util'; -import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; -import { Util } from 'src/shared/utils/util'; -import { PayoutBitcoinService } from 'src/subdomains/supporting/payout/services/payout-bitcoin.service'; -import { PayoutFiroService } from 'src/subdomains/supporting/payout/services/payout-firo.service'; -import { BlockchainRegistryService } from '../../../../integration/blockchain/shared/services/blockchain-registry.service'; - -interface FeeCacheData { - timestamp: Date; - fee: number; -} - -@Injectable() -export class PaymentLinkFeeService implements OnModuleInit { - private readonly logger = new DfxLogger(PaymentLinkFeeService); - - private static readonly MINUTES_5 = 5 * 60; - - private readonly feeCache: Map; - - constructor( - private readonly blockchainRegistryService: BlockchainRegistryService, - private readonly payoutBitcoinService: PayoutBitcoinService, - private readonly payoutFiroService: PayoutFiroService, - ) { - this.feeCache = new Map(); - } - - onModuleInit() { - void this.updateFees(); - } - - // --- JOBS --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.UPDATE_BLOCKCHAIN_FEE }) - async updateFees(): Promise { - if (GetConfig().environment === Environment.LOC) return; - - for (const blockchain of PaymentLinkBlockchains) { - try { - const fee = await this.calculateFee(blockchain); - this.feeCache.set(blockchain, { - timestamp: new Date(), - fee, - }); - } catch (e) { - this.feeCache.delete(blockchain); - this.logger.error(`Failed to get fee for blockchain ${blockchain}:`, e); - } - } - } - - private async calculateFee(blockchain: Blockchain): Promise { - switch (blockchain) { - case Blockchain.BINANCE_PAY: - case Blockchain.KUCOIN_PAY: - case Blockchain.LIGHTNING: - case Blockchain.MONERO: - case Blockchain.ZANO: - case Blockchain.SOLANA: - case Blockchain.TRON: - case Blockchain.CARDANO: - case Blockchain.INTERNET_COMPUTER: - return 0; - - case Blockchain.ETHEREUM: - case Blockchain.SEPOLIA: - case Blockchain.ARBITRUM: - case Blockchain.OPTIMISM: - case Blockchain.BASE: - case Blockchain.GNOSIS: - case Blockchain.POLYGON: - case Blockchain.BINANCE_SMART_CHAIN: { - const client = this.blockchainRegistryService.getEvmClient(blockchain); - return +(await client.getRecommendedGasPrice()); - } - - // The customer minimum is the network's own minimum for an inbound payment to confirm — it - // must NOT include the CPFP/default margin from getSendFeeRate, which exists only for DFX's - // own outbound spends. The value differs per chain because the chains do, but neither carries - // the payout margin. - case Blockchain.BITCOIN: - // Bitcoin fees are user-adjustable and the chain can congest, so use the recommended - // (next-block) rate, which adapts to congestion — floored at the relay minimum so the - // advertised minimum is always relayable. - return Math.max(await this.payoutBitcoinService.getRecommendedFeeRate(), MIN_FEE_RATE_SAT_VB); - - case Blockchain.FIRO: - // Same principle as Bitcoin: Firo's own next-block rate without the payout margin, floored - // at the relay minimum so it stays relayable. The current OCP deposit address is transparent, - // so a Stack Wallet payment is a Spark-spend to it, whose fee sits at the relay floor and - // cannot be raised; Firo does not congest and its node usually returns no estimate, so this - // resolves to the relay floor in practice — exactly what that Spark-spend pays. A dedicated - // relay-floor cap belongs here only once a Spark `sm1…` deposit address is deployed, whose - // protocol-capped fee cannot follow a congestion-adaptive minimum. - return Math.max(await this.payoutFiroService.getRecommendedFeeRate(), MIN_FEE_RATE_SAT_VB); - } - } - - // --- PUBLIC METHODS --- // - async getMinFee(blockchain: Blockchain): Promise { - const cacheData = this.feeCache.get(blockchain); - if (!cacheData) return; - - if (Util.secondsDiff(cacheData.timestamp) > PaymentLinkFeeService.MINUTES_5) return; - - return cacheData.fee; - } -} +import { Injectable, OnModuleInit } from '@nestjs/common'; +import { CronExpression } from '@nestjs/schedule'; +import { Config, CronRole, Environment, GetConfig } from 'src/config/config'; +import { MIN_FEE_RATE_SAT_VB } from 'src/integration/blockchain/bitcoin/services/bitcoin-based-fee.service'; +import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; +import { PaymentLinkBlockchains } from 'src/integration/blockchain/shared/util/blockchain.util'; +import { DfxLogger } from 'src/shared/services/dfx-logger'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; +import { Util } from 'src/shared/utils/util'; +import { PayoutBitcoinService } from 'src/subdomains/supporting/payout/services/payout-bitcoin.service'; +import { PayoutFiroService } from 'src/subdomains/supporting/payout/services/payout-firo.service'; +import { BlockchainRegistryService } from '../../../../integration/blockchain/shared/services/blockchain-registry.service'; + +interface FeeCacheData { + timestamp: Date; + fee: number; +} + +@Injectable() +export class PaymentLinkFeeService implements OnModuleInit { + private readonly logger = new DfxLogger(PaymentLinkFeeService); + + private static readonly MINUTES_5 = 5 * 60; + + private readonly feeCache: Map; + + /** + * The loads currently in flight, one entry per blockchain, so concurrent readers of a cold cache + * share one fetch instead of each starting their own. Without it a burst of quotes for the same + * chain would fire one gas-price call per request: the cache is only written when a load + * RESOLVES, so every request arriving until then sees the same empty entry. + * + * That burst is not hypothetical here. `updateFees` is leased, so among several API processes + * only one wins it per tick and the others rely on this path for a whole minute. + */ + private readonly loading = new Map>(); + + constructor( + private readonly blockchainRegistryService: BlockchainRegistryService, + private readonly payoutBitcoinService: PayoutBitcoinService, + private readonly payoutFiroService: PayoutFiroService, + ) { + this.feeCache = new Map(); + } + + /** + * Warms the cache at boot — but only where something reads it. + * + * `onModuleInit` runs in EVERY process, whatever `CRON_ROLE` says: it is a Nest lifecycle hook, + * not a cron job, so the scope on `updateFees` below does not reach it. Called unconditionally + * it would undo that scope at boot and make the worker do the one thing the scope exists to + * prevent — query gas prices for eight EVM chains plus Bitcoin and Firo to fill a map nothing + * in that process ever reads. + * + * The condition is deliberately about serving requests rather than about which jobs this role + * registers. This cache has exactly one reader, `getMinFee`, and every path to it is a request + * path; a warm cache is worth something where those requests land and nowhere else. Stating it + * that way also keeps the scope table in `runsInThisRole` the single place that maps roles to + * scopes, instead of copying it here where a later change would not reach it. + */ + onModuleInit() { + if (Config.cronRole === CronRole.WORKER) return; + + void this.updateFees(); + } + + // --- JOBS --- // + /** + * Scope Api, not Both: the cache it fills is a field of this service, and the only reader is + * getMinFee below. Following that chain out — createTransferAmount -> createTransferAmounts -> + * createQuote / createPayRequest, plus the Binance webhook handler — every one of them is a + * request path. This job is the only `Api`-scoped cron in the domain, and it is the writer, not + * a reader. None of the Worker jobs in this domain reaches + * the cache: PaymentCronService::forwardDeposits takes its fee rate from BitcoinFeeService, and + * ::processExpiredPayments and ::checkTxConfirmations price nothing — they move a payment out of + * `Pending` and cancel or close what hangs off it. + * + * Both would also break the rule this scope mechanism introduced: a job that runs in every + * process must be harmless twice over, and this one queries gas prices for eight EVM chains + * plus Bitcoin and Firo fee estimates on every tick. In the worker those calls would spend + * quota to produce a value nothing there reads. + */ + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.API, process: Process.UPDATE_BLOCKCHAIN_FEE }) + async updateFees(): Promise { + if (GetConfig().environment === Environment.LOC) return; + + for (const blockchain of PaymentLinkBlockchains) { + try { + const fee = await this.calculateFee(blockchain); + this.feeCache.set(blockchain, { + timestamp: new Date(), + fee, + }); + } catch (e) { + this.feeCache.delete(blockchain); + this.logger.error(`Failed to get fee for blockchain ${blockchain}:`, e); + } + } + } + + private async calculateFee(blockchain: Blockchain): Promise { + switch (blockchain) { + case Blockchain.BINANCE_PAY: + case Blockchain.KUCOIN_PAY: + case Blockchain.LIGHTNING: + case Blockchain.MONERO: + case Blockchain.ZANO: + case Blockchain.SOLANA: + case Blockchain.TRON: + case Blockchain.CARDANO: + case Blockchain.INTERNET_COMPUTER: + return 0; + + case Blockchain.ETHEREUM: + case Blockchain.SEPOLIA: + case Blockchain.ARBITRUM: + case Blockchain.OPTIMISM: + case Blockchain.BASE: + case Blockchain.GNOSIS: + case Blockchain.POLYGON: + case Blockchain.BINANCE_SMART_CHAIN: { + const client = this.blockchainRegistryService.getEvmClient(blockchain); + return +(await client.getRecommendedGasPrice()); + } + + // The customer minimum is the network's own minimum for an inbound payment to confirm — it + // must NOT include the CPFP/default margin from getSendFeeRate, which exists only for DFX's + // own outbound spends. The value differs per chain because the chains do, but neither carries + // the payout margin. + case Blockchain.BITCOIN: + // Bitcoin fees are user-adjustable and the chain can congest, so use the recommended + // (next-block) rate, which adapts to congestion — floored at the relay minimum so the + // advertised minimum is always relayable. + return Math.max(await this.payoutBitcoinService.getRecommendedFeeRate(), MIN_FEE_RATE_SAT_VB); + + case Blockchain.FIRO: + // Same principle as Bitcoin: Firo's own next-block rate without the payout margin, floored + // at the relay minimum so it stays relayable. The current OCP deposit address is transparent, + // so a Stack Wallet payment is a Spark-spend to it, whose fee sits at the relay floor and + // cannot be raised; Firo does not congest and its node usually returns no estimate, so this + // resolves to the relay floor in practice — exactly what that Spark-spend pays. A dedicated + // relay-floor cap belongs here only once a Spark `sm1…` deposit address is deployed, whose + // protocol-capped fee cannot follow a congestion-adaptive minimum. + return Math.max(await this.payoutFiroService.getRecommendedFeeRate(), MIN_FEE_RATE_SAT_VB); + } + } + + // --- PUBLIC METHODS --- // + /** + * Loads on demand when the cache has nothing usable, rather than answering `undefined`. + * + * CONTRIBUTING: "A cache read in a request path must load on demand … A cron job may refresh + * it, but must not be the only thing filling it." The job above refreshes; this is the load. + * + * Before the split that distinction did not bite: one process ran the refresh and served the + * requests, so a filled cache and a reading request were the same process. Now the refresh is + * `Api`-scoped AND leased, so among several API processes only one wins it per tick — the + * others would serve `undefined` for a fee they could have fetched, and `createQuote` would + * price without it. + * + * Only the blockchain that was asked for is loaded, not the whole set: a request pays for what + * it needs, and the job stays the thing that keeps the rest warm. + */ + async getMinFee(blockchain: Blockchain): Promise { + const cacheData = this.feeCache.get(blockchain); + const usable = cacheData && Util.secondsDiff(cacheData.timestamp) <= PaymentLinkFeeService.MINUTES_5; + if (usable) return cacheData.fee; + + // The same guard the job carries, for the same reason. On LOC there are no node connections to + // ask, so `updateFees` never fills the cache and this path would run into the timeout of every + // one of those calls on every request. Answering `undefined` is what a local environment + // returned before this method loaded anything, and the callers already handle it. + if (GetConfig().environment === Environment.LOC) return undefined; + + try { + return await this.loadFee(blockchain); + } catch (e) { + // Same shape as the job's own failure handling: a fee source that cannot be reached leaves + // the caller without a minimum, which is what it would have had anyway. Logged rather than + // thrown, so one unreachable chain does not fail a quote for the others. + this.logger.error(`Failed to load fee for blockchain ${blockchain} on demand:`, e); + + return undefined; + } + } + + // --- HELPER METHODS --- // + /** + * One load per blockchain at a time; concurrent callers await the one already running. + * + * The entry is removed in `finally`, before the promise is handed out. A failed load therefore + * leaves nothing behind that a later call would await forever, and the next caller retries + * rather than inheriting the failure — the cache is a fee that expires, not a decision. + */ + private loadFee(blockchain: Blockchain): Promise { + const running = this.loading.get(blockchain); + if (running) return running; + + const load = (async () => { + try { + const fee = await this.calculateFee(blockchain); + this.feeCache.set(blockchain, { timestamp: new Date(), fee }); + + return fee; + } finally { + this.loading.delete(blockchain); + } + })(); + + this.loading.set(blockchain, load); + + return load; + } +} diff --git a/src/subdomains/core/payment-link/services/payment-link-payment.service.ts b/src/subdomains/core/payment-link/services/payment-link-payment.service.ts index e7f223f2d9..0c1d3c6667 100644 --- a/src/subdomains/core/payment-link/services/payment-link-payment.service.ts +++ b/src/subdomains/core/payment-link/services/payment-link-payment.service.ts @@ -1,461 +1,852 @@ -import { BadRequestException, ConflictException, Injectable, NotFoundException } from '@nestjs/common'; -import { Observable, Subject } from 'rxjs'; -import { Config, Environment } from 'src/config/config'; -import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; -import { BlockchainRegistryService } from 'src/integration/blockchain/shared/services/blockchain-registry.service'; -import { LnurlpInvoiceDto } from 'src/integration/lightning/dto/lnurlp.dto'; -import { LightningHelper } from 'src/integration/lightning/lightning-helper'; -import { FiatService } from 'src/shared/models/fiat/fiat.service'; -import { AsyncMap } from 'src/shared/utils/async-map'; -import { Util } from 'src/shared/utils/util'; -import { C2BWebhookResult } from 'src/subdomains/core/payment-link/share/c2b-payment-link.provider'; -import { CryptoInput } from 'src/subdomains/supporting/payin/entities/crypto-input.entity'; -import { IsNull, LessThan } from 'typeorm'; -import { isSellRoute } from '../../sell-crypto/route/sell.entity'; -import { CreatePaymentLinkPaymentDto } from '../dto/create-payment-link-payment.dto'; -import { PaymentLinkEvmPaymentDto, PaymentLinkHexResultDto, TransferInfo } from '../dto/payment-link.dto'; -import { PaymentRequestMapper } from '../dto/payment-request.mapper'; -import { UpdatePaymentLinkPaymentDto } from '../dto/update-payment-link-payment.dto'; -import { PaymentDevice, PaymentLinkPayment } from '../entities/payment-link-payment.entity'; -import { PaymentLink } from '../entities/payment-link.entity'; -import { PaymentQuote } from '../entities/payment-quote.entity'; -import { - PaymentLinkMode, - PaymentLinkPaymentMode, - PaymentLinkPaymentStatus, - PaymentLinkStatus, - PaymentQuoteFinalStates, - PaymentQuoteStatus, - PaymentQuoteTxStates, -} from '../enums'; -import { PaymentLinkPaymentRepository } from '../repositories/payment-link-payment.repository'; -import { PaymentActivationService } from './payment-activation.service'; -import { PaymentQuoteService } from './payment-quote.service'; -import { PaymentWebhookService } from './payment-webhook.service'; - -@Injectable() -export class PaymentLinkPaymentService { - private readonly paymentWaitMap = new AsyncMap(this.constructor.name); - private readonly deviceActivationSubject = new Subject(); - - constructor( - private readonly fiatService: FiatService, - private readonly paymentLinkPaymentRepo: PaymentLinkPaymentRepository, - private readonly paymentWebhookService: PaymentWebhookService, - private readonly paymentQuoteService: PaymentQuoteService, - private readonly paymentActivationService: PaymentActivationService, - private readonly blockchainRegistryService: BlockchainRegistryService, - ) {} - - getDeviceActivationObservable(): Observable { - return this.deviceActivationSubject.asObservable(); - } - - // --- JOBS --- // - async processExpiredPayments(): Promise { - const maxDate = Util.secondsBefore(Config.payment.timeoutDelay); - - const pendingPayments = await this.paymentLinkPaymentRepo.find({ - where: { - status: PaymentLinkPaymentStatus.PENDING, - expiryDate: LessThan(maxDate), - }, - relations: { link: true }, - }); - - for (const payment of pendingPayments) { - await this.expirePayment(payment); - } - } - - async expirePayment(payment: PaymentLinkPayment): Promise { - await this.doSave(payment.expire(), true); - await this.cancelQuotesForPayment(payment); - } - - async checkTxConfirmations(): Promise { - const confirmingQuotes = await this.paymentQuoteService.getConfirmingQuotes(); - - for (const quote of confirmingQuotes) { - const blockchain = quote.txBlockchain; - - if (blockchain) { - const client = this.blockchainRegistryService.getClient(blockchain); - const isTxComplete = await client.isTxComplete(quote.txId, Config.payment.minConfirmations(blockchain)); - - if (isTxComplete) { - await this.paymentQuoteService.saveFinallyConfirmed(quote); - await this.handleQuoteChange(quote.payment, quote); - } - } - } - } - - // --- CRUD --- // - - async updatePayment(id: number, dto: UpdatePaymentLinkPaymentDto): Promise { - const entity = await this.paymentLinkPaymentRepo.findOneBy({ id }); - if (!entity) throw new NotFoundException('Payment not found'); - - return this.paymentLinkPaymentRepo.save(Object.assign(entity, dto)); - } - - async getPendingPaymentByUniqueId(uniqueId: string): Promise { - return this.paymentLinkPaymentRepo.findOne({ - where: [ - { - link: { uniqueId }, - status: PaymentLinkPaymentStatus.PENDING, - }, - { - uniqueId, - status: PaymentLinkPaymentStatus.PENDING, - }, - ], - relations: { - link: { route: { deposit: true, user: { userData: { organization: true } } } }, - }, - }); - } - - // externalPaymentId is a merchant-supplied reconciliation identifier and is NOT unique across - // merchants; scope the lookup to a link the caller has already been authorized against, or the - // response leaks foreign merchants' payment records (BUG-1289). - async getPaymentByExternalId(linkId: number, externalPaymentId: string): Promise { - return this.paymentLinkPaymentRepo.findOne({ - where: { externalId: externalPaymentId, link: { id: linkId } }, - }); - } - - async getMostRecentPayment(uniqueId: string): Promise { - return this.paymentLinkPaymentRepo.findOne({ - where: [ - { - link: { uniqueId: uniqueId }, - }, - { - uniqueId: uniqueId, - }, - ], - order: { updated: 'DESC' }, - }); - } - - async getMostRecentPayments(linkIds: number[]): Promise { - if (!linkIds.length) return []; - - return this.paymentLinkPaymentRepo - .createQueryBuilder('plp') - .innerJoin( - (qb) => - qb - .select('plp2."linkId"', 'linkId') - .addSelect('MAX(plp2.id)', 'maxId') - .from(PaymentLinkPayment, 'plp2') - .groupBy('plp2."linkId"'), - 'latest', - 'latest."linkId" = plp."linkId" AND latest."maxId" = plp.id', - ) - .innerJoinAndSelect('plp.currency', 'currency') - .innerJoinAndSelect('plp.link', 'link') - .where('link.id IN (:...ids)', { ids: linkIds }) - .getMany(); - } - - // --- HANDLE WAITS --- // - async waitForPayment(payment: PaymentLinkPayment): Promise { - return this.paymentWaitMap.wait(payment.id, 0); - } - - async handleBinanceWaiting(result: C2BWebhookResult): Promise { - const { qrContent, referId } = result.metadata; - - const lnurl = new URL(qrContent).searchParams.get('lightning'); - const uniqueId = LightningHelper.decodeLnurl(lnurl).split('/').at(-1); - const payment = await this.getPendingPaymentByUniqueId(uniqueId); - if (!payment) throw new NotFoundException('Payment not found'); - - const quote = await this.paymentQuoteService.createQuote(payment.link.defaultStandard, payment); - const transferAmount = JSON.parse(quote.transferAmounts).find((t) => t.method === Blockchain.BINANCE_PAY); - if (!transferAmount?.assets.length) throw new NotFoundException('Transfer amount not found'); - - const transferInfo: TransferInfo = { - asset: transferAmount.assets[0].asset, - amount: transferAmount.assets[0].amount, - method: Blockchain.BINANCE_PAY, - quoteUniqueId: quote.uniqueId, - referId, - }; - - await this.createActivationRequest(payment.uniqueId, transferInfo); - } - - async createPayment(paymentLink: PaymentLink, dto: CreatePaymentLinkPaymentDto): Promise { - if (paymentLink.status !== PaymentLinkStatus.ACTIVE) throw new BadRequestException('Payment link is not active'); - - const pendingPayment = paymentLink.payments.some((p) => p.status === PaymentLinkPaymentStatus.PENDING); - if (pendingPayment) - throw new ConflictException('There is already a pending payment for the specified payment link'); - - if (paymentLink.mode === PaymentLinkMode.SINGLE) { - const hasPreviousPayment = await this.paymentLinkPaymentRepo.existsBy({ - link: { uniqueId: paymentLink.uniqueId }, - }); - if (hasPreviousPayment) throw new ConflictException('Single payment link can only have one payment'); - } - - if (dto.externalId) { - const exists = await this.paymentLinkPaymentRepo.existsBy({ - externalId: dto.externalId, - link: { id: paymentLink.id }, - }); - if (exists) throw new ConflictException('Payment already exists'); - } - - if (isSellRoute(paymentLink.route) && dto.currency && dto.currency !== paymentLink.route.fiat.name) - throw new BadRequestException('Payment currency mismatch'); - - const currency = isSellRoute(paymentLink.route) - ? paymentLink.route.fiat - : await this.fiatService.getFiatByName(dto.currency ?? 'CHF'); - - const payment = this.paymentLinkPaymentRepo.create({ - amount: dto.amount, - externalId: dto.externalId, - note: dto.note, - expiryDate: dto.expiryDate ?? Util.secondsAfter(paymentLink.configObj.paymentTimeout), - mode: dto.mode ?? PaymentLinkPaymentMode.SINGLE, - currency, - uniqueId: Util.createUniqueId(Config.prefixes.paymentLinkPaymentUidPrefix), - status: PaymentLinkPaymentStatus.PENDING, - link: paymentLink, - }); - - const savedPayment = await this.doSave(payment, false); - - // auto confirm (DEV only) - if (Config.environment !== Environment.PRD && paymentLink.configObj.autoConfirmSecs != null) { - setTimeout(async () => { - if (payment.amount === 0.01) { - payment.cancel(); - } else { - payment.complete(); - } - await this.doSave(payment, true); - }, paymentLink.configObj.autoConfirmSecs * 1000); - } - - // expiry timers - const scanTimeout = paymentLink.configObj.scanTimeout; - if (scanTimeout) { - setTimeout(() => this.expirePaymentIfPending(payment.id, true), scanTimeout * 1000); - } - - const paymentExpiry = Util.secondsAfter(Config.payment.timeoutDelay, payment.expiryDate); - if (Util.minutesDiff(new Date(), paymentExpiry) <= 60) { - const paymentTimeout = paymentExpiry.getTime() - new Date().getTime(); - setTimeout(() => this.expirePaymentIfPending(payment.id, false), paymentTimeout); - } - - return savedPayment; - } - - private async expirePaymentIfPending(id: number, ignoreWithQuote: boolean): Promise { - const pendingPayment = await this.paymentLinkPaymentRepo.findOne({ - where: { - id, - status: PaymentLinkPaymentStatus.PENDING, - quotes: { id: ignoreWithQuote ? IsNull() : undefined }, - }, - relations: { link: true }, - }); - - if (pendingPayment) await this.expirePayment(pendingPayment); - } - - async confirmPayment(payment: PaymentLinkPayment): Promise { - if (payment.status !== PaymentLinkPaymentStatus.COMPLETED) - throw new BadRequestException('Payment is not completed'); - - await this.paymentLinkPaymentRepo.update(payment.id, { isConfirmed: true }); - } - - async cancelByLink(paymentLink: PaymentLink): Promise { - const pendingPayment = paymentLink.payments.find((p) => p.status === PaymentLinkPaymentStatus.PENDING); - if (!pendingPayment) throw new NotFoundException('No pending payment found'); - - pendingPayment.link = paymentLink; - - await this.cancelByPayment(pendingPayment); - - return paymentLink; - } - - async cancelByPayment(payment: PaymentLinkPayment): Promise { - await this.doSave(payment.cancel(), true); - await this.cancelQuotesForPayment(payment); - } - - async deletePayment(payment: PaymentLinkPayment): Promise { - if (payment.status === PaymentLinkPaymentStatus.COMPLETED) - throw new BadRequestException('PaymentLinkPayment is already completed, cannot be deleted'); - - for (const quote of payment.quotes) { - await this.paymentQuoteService.deleteQuote(quote); - } - - for (const activation of payment.activations) { - await this.paymentActivationService.deleteActivation(activation); - } - - await this.paymentLinkPaymentRepo.delete(payment.id); - } - - private async cancelQuotesForPayment(payment: PaymentLinkPayment): Promise { - await this.paymentQuoteService.cancelAllForPayment(payment.id); - await this.paymentActivationService.closeAllForPayment(payment.id); - } - - // --- HANDLE CALLBACKS --- // - async createActivationRequest( - uniqueId: string, - transferInfo: TransferInfo, - ): Promise { - const pendingPayment = await this.getPendingPaymentByUniqueId(uniqueId); - if (!pendingPayment) throw new NotFoundException(`Pending payment not found by id ${uniqueId}`); - - const activation = await this.paymentActivationService.doCreateRequest(pendingPayment, transferInfo); - return PaymentRequestMapper.toPaymentRequest(activation); - } - - async handleHexPayment(uniqueId: string, transferInfo: TransferInfo): Promise { - const pendingPayment = await this.getPendingPaymentByUniqueId(uniqueId); - if (!pendingPayment) throw new NotFoundException(`Pending payment not found by id ${uniqueId}`); - - const quote = await this.paymentQuoteService.executeHexPayment(transferInfo); - await this.handleQuoteChange(pendingPayment, quote); - - if (quote.status === PaymentQuoteStatus.TX_FAILED) - throw new BadRequestException(`Failed to handle hex payment ${uniqueId}: ${quote.errorMessage}`); - - return { txId: quote.txId }; - } - - // --- HANDLE INPUTS --- // - async getPaymentQuoteByFailedCryptoInput(cryptoInput: CryptoInput): Promise { - const quote = await this.paymentQuoteService.getQuoteByTxId(cryptoInput.address.blockchain, cryptoInput.inTxId, [ - PaymentQuoteStatus.TX_MEMPOOL, - PaymentQuoteStatus.TX_BLOCKCHAIN, - PaymentQuoteStatus.TX_COMPLETED, - ]); - if (!quote) return null; - - if (quote.status === PaymentQuoteStatus.TX_MEMPOOL) { - await this.handleBlockchainConfirmed(quote, cryptoInput); - } - - return quote; - } - - async getPaymentQuoteByCryptoInput(cryptoInput: CryptoInput): Promise { - const quote = await this.getQuoteForInput(cryptoInput); - if (!quote) throw new Error(`No matching quote found`); - - await this.handleBlockchainConfirmed(quote, cryptoInput); - - return quote; - } - - private async handleBlockchainConfirmed(quote: PaymentQuote, cryptoInput: CryptoInput): Promise { - await this.paymentQuoteService.saveBlockchainConfirmed(quote, cryptoInput.address.blockchain, cryptoInput.inTxId); - - const payment = await this.paymentLinkPaymentRepo.findOne({ - where: { id: quote.payment.id }, - relations: { link: { route: { user: { userData: { organization: true } } } } }, - }); - - await this.handleQuoteChange(payment, quote); - } - - private async getQuoteForInput(cryptoInput: CryptoInput): Promise { - const quote = [Blockchain.LIGHTNING, Blockchain.BINANCE_PAY, Blockchain.KUCOIN_PAY].includes( - cryptoInput.address.blockchain, - ) - ? await this.getQuoteByActivation(cryptoInput.address.blockchain, cryptoInput.inTxId) - : await this.getQuoteByTx(cryptoInput.address.blockchain, cryptoInput.inTxId); - - if (quote) return quote; - - return this.paymentQuoteService.getQuoteByAsset(cryptoInput.asset, cryptoInput.amount); - } - - private async getQuoteByActivation(txBlockchain: Blockchain, txId: string): Promise { - const activation = await this.paymentActivationService.getActivationByTxId(txId); - if (!activation) return null; - - const quote = activation.quote; - if (quote && !quote.txId) await this.paymentQuoteService.saveTransaction(quote, txBlockchain, txId); - - return quote; - } - - private async getQuoteByTx(txBlockchain: Blockchain, txId: string): Promise { - return this.paymentQuoteService.getQuoteByTxId(txBlockchain, txId, [ - PaymentQuoteStatus.TX_RECEIVED, - PaymentQuoteStatus.TX_MEMPOOL, - PaymentQuoteStatus.TX_BLOCKCHAIN, - ]); - } - - private async handleQuoteChange(payment: PaymentLinkPayment, quote: PaymentQuote): Promise { - // close activations - if (PaymentQuoteFinalStates.includes(quote.status)) - if (payment.mode === PaymentLinkPaymentMode.SINGLE) { - await this.paymentActivationService.closeAllForPayment(payment.id); - } else { - await this.paymentActivationService.closeAllForQuote(quote.id); - } - - if (payment.status !== PaymentLinkPaymentStatus.PENDING) return; - - // update payment status - const { minCompletionStatus } = payment.link.configObj; - - const isPaymentComplete = - PaymentQuoteTxStates.indexOf(quote.status) >= PaymentQuoteTxStates.indexOf(minCompletionStatus); - if (isPaymentComplete) { - payment.txCount = await this.paymentQuoteService.getCompletedQuoteCount(payment, minCompletionStatus); - - if (payment.mode === PaymentLinkPaymentMode.SINGLE) payment.complete(); - - await this.doSave(payment, true); - } - } - - private async doSave(payment: PaymentLinkPayment, isPaymentDone: boolean): Promise { - const savedPayment = await this.paymentLinkPaymentRepo.save(payment); - - if (savedPayment.link.webhookUrl) await this.sendWebhook(savedPayment); - - if (isPaymentDone) { - this.paymentWaitMap.resolve(savedPayment.id, savedPayment); - if (payment.device) this.deviceActivationSubject.next(payment.device); - } - - return savedPayment; - } - - private async sendWebhook(payment: PaymentLinkPayment): Promise { - const paymentForWebhook = await this.paymentLinkPaymentRepo.findOne({ - where: { uniqueId: payment.uniqueId }, - relations: { - link: { route: { user: { userData: { organization: true } } } }, - }, - }); - - const paymentLink = paymentForWebhook.link; - paymentLink.payments = [paymentForWebhook]; - - await this.paymentWebhookService.sendWebhook(paymentLink); - } -} +import { BadRequestException, ConflictException, Injectable, NotFoundException } from '@nestjs/common'; +import { Config, Environment } from 'src/config/config'; +import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; +import { BlockchainRegistryService } from 'src/integration/blockchain/shared/services/blockchain-registry.service'; +import { LnurlpInvoiceDto } from 'src/integration/lightning/dto/lnurlp.dto'; +import { LightningHelper } from 'src/integration/lightning/lightning-helper'; +import { FiatService } from 'src/shared/models/fiat/fiat.service'; +import { AsyncMap } from 'src/shared/utils/async-map'; +import { Util } from 'src/shared/utils/util'; +import { C2BWebhookResult } from 'src/subdomains/core/payment-link/share/c2b-payment-link.provider'; +import { CryptoInput } from 'src/subdomains/supporting/payin/entities/crypto-input.entity'; +import { EntityManager, In, IsNull, LessThan, MoreThan, Not } from 'typeorm'; +import { isSellRoute } from '../../sell-crypto/route/sell.entity'; +import { CreatePaymentLinkPaymentDto } from '../dto/create-payment-link-payment.dto'; +import { PaymentLinkEvmPaymentDto, PaymentLinkHexResultDto, TransferInfo } from '../dto/payment-link.dto'; +import { PaymentRequestMapper } from '../dto/payment-request.mapper'; +import { UpdatePaymentLinkPaymentDto } from '../dto/update-payment-link-payment.dto'; +import { PaymentDevice, PaymentLinkPayment } from '../entities/payment-link-payment.entity'; +import { PaymentLink } from '../entities/payment-link.entity'; +import { PaymentQuote } from '../entities/payment-quote.entity'; +import { + PaymentLinkMode, + PaymentLinkPaymentMode, + PaymentLinkPaymentStatus, + PaymentLinkStatus, + PaymentQuoteFinalStates, + PaymentQuoteStatus, + PaymentQuoteTxStates, +} from '../enums'; +import { PaymentLinkPaymentRepository } from '../repositories/payment-link-payment.repository'; +import { PaymentActivationService } from './payment-activation.service'; +import { PaymentQuoteService } from './payment-quote.service'; +import { PaymentWebhookService } from './payment-webhook.service'; + +/** + * How long a caller of `waitForPayment` is held before it is answered with the state on hand. + * + * A server-side long poll without an upper bound leaks by construction: a client that hangs up + * says nothing the server can hear, so its entry would sit in the maps below until the process + * restarts. The bound is what gives the entry an owner — with it, the waiter itself clears the + * entry on the way out, whichever way it leaves. + * + * The endpoints answering from this do not change shape when it elapses; they answer with the + * payment as it stands, which for a caller that is still there means "not yet, ask again". + * + * The bound belongs to the ENTRY, not to each caller: AsyncMap hands a second waiter on the same + * payment the first one's promise, so it is answered after the REMAINDER of that timer, not after + * a fresh 60 s. For a payment watched from two ends — the terminal and the customer's wallet — + * that means one of them polls a little more often than the number above suggests. + */ +const PAYMENT_WAIT_TIMEOUT_SECONDS = 60; + +/** + * A device this process holds at least one open websocket connection for. + * + * Only the identity: what a device is owed follows from the payments themselves, not from when it + * happened to connect. A device that reconnects is the same device, and the delivery record below + * outlives the connection precisely so that it is treated as one — within the span the read + * covers. What has aged out of that span is owed to nobody any more. + */ +export interface ConnectedDevice { + id: string; +} + +/** + * How long after a payment can no longer expire the delivery read below still asks for it. + * + * The read selects on `expiryDate`, and the reason is which writes can move a column. `updated` is + * stamped by every write; `expiryDate` is given at insert and never mutated afterwards. That is the + * whole difference. Any predicate over a column a later write can move is able to carry a row OUT + * of the read before the read has seen it, and two rounds of review found the same failure twice + * that way: a mark advanced past a row that had not committed, and then a span against the present + * that a transaction outliving it walks a row straight past. + * + * What the immutable column buys, precisely: no WRITE can move a row out of the read. A late + * commit makes it appear later, never skip, because its place was fixed at insert. What it does + * NOT buy: the read still ends somewhere, so a transition that happens after that end is missed + * like any other. The two are different failures — the first was a property of the predicate and + * is gone; the second is a question of how far the span reaches, and is answered below. + * + * So this is not a window a payment has to be delivered within. It is how far past a payment's own + * end the read keeps asking, and it is measured from `expiryDate` rather than from now: + * `processExpiredPayments` expires a payment at `expiryDate` plus `Config.payment.timeoutDelay`, so + * the span has to outlast that delay for the expiry transition itself to still be read — which is + * why the cutoff below ADDS the configured delay instead of assuming it away. + * + * The hour is chosen against the thing that actually delays the transition: the worker being gone. + * `processExpiredPayments` runs there, so a worker that is down does not expire anything, and the + * transitions arrive in a burst when it returns. Ten minutes covered a deploy; it did not cover an + * outage, and the alert on a silent worker only fires after seventeen. An hour outlasts both, and + * costs a longer read of a handful of rows per connected device. + */ +const DEVICE_DELIVERY_GRACE_SECONDS = 3600; + +/** + * What this process has delivered to one device: per payment, the wait state last sent for it and + * the `expiryDate` that decides how long the entry is kept. Never a record of who is connected — + * see `connectedDevices`. + */ +type DeviceDeliveries = Map; + +@Injectable() +export class PaymentLinkPaymentService { + private readonly paymentWaitMap = new AsyncMap(this.constructor.name); + private readonly waitStates = new Map(); + /** + * Where a device command is HANDED OVER, and what says whether it got there. Empty until the + * gateway sets it, which is the honest answer for a process holding no websocket at all. + */ + private deviceSink: (device: PaymentDevice) => boolean = () => false; + private readonly deviceDeliveries = new Map(); + + /** + * Where the connected devices are READ from — set once by PaymentLinkGateway, which owns the + * sockets. + * + * A device is connected for exactly as long as the gateway holds a socket for it, so the socket + * map is the only thing that knows. Reading through to it beats having the gateway report + * connects and disconnects into a second map here: a mirrored register can drift from what it + * mirrors, and every way it drifted was a defect — an entry left behind when no close event + * arrived, a count decremented twice by a close path taken twice, an entry with no counterpart. + * A derived register has no second copy to get out of step with. + * + * Empty until the gateway sets it, which is also the honest answer for a process that accepts no + * websocket connections at all. + */ + private connectedDevices: () => ConnectedDevice[] = () => []; + + constructor( + private readonly fiatService: FiatService, + private readonly paymentLinkPaymentRepo: PaymentLinkPaymentRepository, + private readonly paymentWebhookService: PaymentWebhookService, + private readonly paymentQuoteService: PaymentQuoteService, + private readonly paymentActivationService: PaymentActivationService, + private readonly blockchainRegistryService: BlockchainRegistryService, + ) {} + + /** + * Points the delivery at the gateway's sockets; see `deliverToDevice` for why it returns a + * boolean. Called once, by PaymentLinkGateway, which is the only thing that holds them. + * + * Replaces an RxJS subject the gateway subscribed to. A subject carries a value one way and + * swallows what the subscriber does with it — including a `send` that threw — and this delivery + * has to know whether the command arrived: the record it keeps is a record of what a device HAS + * been told. A subject cannot answer that question, so it was the wrong shape here. + */ + useDeviceSink(sink: (device: PaymentDevice) => boolean): void { + this.deviceSink = sink; + } + + // --- JOBS --- // + async processExpiredPayments(): Promise { + const maxDate = Util.secondsBefore(Config.payment.timeoutDelay); + + const pendingPayments = await this.paymentLinkPaymentRepo.find({ + where: { + status: PaymentLinkPaymentStatus.PENDING, + expiryDate: LessThan(maxDate), + }, + relations: { link: true }, + }); + + for (const payment of pendingPayments) { + await this.expirePayment(payment); + } + } + + async expirePayment(payment: PaymentLinkPayment): Promise { + const taken = await this.takePendingTransition(payment, PaymentLinkPaymentStatus.EXPIRED, (manager) => + this.cancelQuotesForPayment(payment.id, manager), + ); + if (!taken) return; + + await this.doSave(payment.expire(), true); + } + + /** + * Moves a payment out of `Pending` together with the database effects that belong to that + * transition, and answers whether THIS caller is the one that moved it. Everything that follows + * — the merchant webhook, the deliveries in `doSave` — belongs to the caller that gets `true`. + * + * The read-then-write this replaces was safe while everything ran in one process. It is not any + * more: `processExpiredPayments` is a `Worker` job, while the expiry timers `createPayment` arms + * stay in the process that served the request, so two processes can read the same row as + * `Pending` and both act on it. A lock would not help, because they are different processes, and + * a lease would not either, because the request paths below reach the same transition without + * going through a job at all. + * + * The `status` in the criteria is what decides it: the database lets exactly one statement past + * `Pending` and reports it as the affected row, and every other caller gets nothing. That holds + * for any number of processes and for every path into the transition, which is why it sits here + * rather than at the call sites. + * + * `effects` runs in the same transaction as that statement, so the row leaves `Pending` only if + * they leave with it. A statement committing on its own would be worse than the double run it + * prevents: a caller dying between the two would leave a payment out of `Pending` with its + * quotes still open, and nothing looks for that — `processExpiredPayments` asks for `Pending` + * and would never see the row again. Rolled back, the payment stays exactly where the next run + * of the job, or of the timer, picks it up. + * + * What stays outside is what a transaction must not hold open: the merchant webhook and the + * process-local deliveries in `doSave`. Those cost a notification when they are lost, not a row + * that no one reconciles — and the webhook is best-effort by construction (see + * `PaymentWebhookService.sendWebhook`). + */ + private async takePendingTransition( + payment: PaymentLinkPayment, + status: PaymentLinkPaymentStatus, + effects: (manager: EntityManager) => Promise, + ): Promise { + return this.paymentLinkPaymentRepo.manager.transaction(async (manager) => { + const { affected } = await manager.update( + PaymentLinkPayment, + { id: payment.id, status: PaymentLinkPaymentStatus.PENDING }, + { status }, + ); + if (!affected) return false; + + await effects(manager); + + return true; + }); + } + + async checkTxConfirmations(): Promise { + const confirmingQuotes = await this.paymentQuoteService.getConfirmingQuotes(); + + for (const quote of confirmingQuotes) { + const blockchain = quote.txBlockchain; + + if (blockchain) { + const client = this.blockchainRegistryService.getClient(blockchain); + const isTxComplete = await client.isTxComplete(quote.txId, Config.payment.minConfirmations(blockchain)); + + if (isTxComplete) { + await this.paymentQuoteService.saveFinallyConfirmed(quote); + await this.handleQuoteChange(quote.payment, quote); + } + } + } + } + + // --- CRUD --- // + + async updatePayment(id: number, dto: UpdatePaymentLinkPaymentDto): Promise { + const entity = await this.paymentLinkPaymentRepo.findOneBy({ id }); + if (!entity) throw new NotFoundException('Payment not found'); + + return this.paymentLinkPaymentRepo.save(Object.assign(entity, dto)); + } + + async getPendingPaymentByUniqueId(uniqueId: string): Promise { + return this.paymentLinkPaymentRepo.findOne({ + where: [ + { + link: { uniqueId }, + status: PaymentLinkPaymentStatus.PENDING, + }, + { + uniqueId, + status: PaymentLinkPaymentStatus.PENDING, + }, + ], + relations: { + link: { route: { deposit: true, user: { userData: { organization: true } } } }, + }, + }); + } + + // externalPaymentId is a merchant-supplied reconciliation identifier and is NOT unique across + // merchants; scope the lookup to a link the caller has already been authorized against, or the + // response leaks foreign merchants' payment records (BUG-1289). + async getPaymentByExternalId(linkId: number, externalPaymentId: string): Promise { + return this.paymentLinkPaymentRepo.findOne({ + where: { externalId: externalPaymentId, link: { id: linkId } }, + }); + } + + async getMostRecentPayment(uniqueId: string): Promise { + return this.paymentLinkPaymentRepo.findOne({ + where: [ + { + link: { uniqueId: uniqueId }, + }, + { + uniqueId: uniqueId, + }, + ], + order: { updated: 'DESC' }, + }); + } + + async getMostRecentPayments(linkIds: number[]): Promise { + if (!linkIds.length) return []; + + return this.paymentLinkPaymentRepo + .createQueryBuilder('plp') + .innerJoin( + (qb) => + qb + .select('plp2."linkId"', 'linkId') + .addSelect('MAX(plp2.id)', 'maxId') + .from(PaymentLinkPayment, 'plp2') + .groupBy('plp2."linkId"'), + 'latest', + 'latest."linkId" = plp."linkId" AND latest."maxId" = plp.id', + ) + .innerJoinAndSelect('plp.currency', 'currency') + .innerJoinAndSelect('plp.link', 'link') + .where('link.id IN (:...ids)', { ids: linkIds }) + .getMany(); + } + + // --- HANDLE WAITS --- // + + /** + * Both delivery channels of this service are process-local: the map behind this method and the + * sink behind `useDeviceSink`. A caller therefore only ever hears from the + * process holding its connection, while the jobs that move a payment forward run in one process + * (`CronScope.WORKER`, which the deployment runs once and the cron lease keeps to one claim). + * + * `deliverPaymentUpdates` below bridges the two. It reads the persisted state of the payments + * THIS process is waiting on and releases them here, so the delivery no longer depends on which + * process did the writing. `doSave` still delivers directly, which keeps the single-process + * case (`CRON_ROLE=all`) as immediate as it is today; the job is the catch-up path for every + * other process. + */ + async waitForPayment(payment: PaymentLinkPayment): Promise { + // The state to compare against, taken before the wait: what the caller is waiting for is a + // change from it, not a fixed target state (see PaymentLinkPayment.waitState). + if (!this.waitStates.has(payment.id)) this.waitStates.set(payment.id, payment.waitState); + + try { + return await this.paymentWaitMap.wait(payment.id, PAYMENT_WAIT_TIMEOUT_SECONDS * 1000); + } catch { + // The wait elapsed. Answer with the payment AS THE CALLER HANDED IT IN rather than fail: + // "nothing was observed within this window" is an answer to what it asked, and the caller + // polls again. + // + // Deliberately not re-read here, although that would close a gap of up to one tick: a change + // in the last 15 s before the timeout has not been picked up yet, so this can answer + // `Pending` for a payment that just completed. Re-reading would cost one query PER TIMING-OUT + // CALLER, and they time out together — a shop with many terminals would turn one batched read + // every 15 s into a burst of single-row reads. The tick path already answers all of them with + // one query, and the next poll is at most a round trip away. + return payment; + } finally { + // The waiter owns its entry. `resolveWaiters` clears it on the delivery path; this clears it + // on every other one — which is the path that used to have no owner at all. Guarded on the + // wait map so a wait registered in the meantime keeps the state it is comparing against. + if (!this.paymentWaitMap.has(payment.id)) this.waitStates.delete(payment.id); + } + } + + /** + * Points the delivery below at the gateway's socket map; see `connectedDevices`. Called once, by + * PaymentLinkGateway, which is the only thing that knows what is connected here. + */ + useDeviceSource(source: () => ConnectedDevice[]): void { + this.connectedDevices = source; + } + + /** + * Delivers what this process is waiting for, from the database rather than from the job that + * wrote it. Writes nothing and calls nothing outside the process, so it is safe to run in every + * process at once — which is what `CronScope.BOTH` requires and what makes it exempt from the + * lease that confines the writing jobs to one process. + * + * Both halves are bounded by what this process actually holds: with no caller waiting and no + * device connected they touch the database not at all. + * + * That it reads on a tick rather than subscribing is the choice this makes against CONTRIBUTING's + * "initial fetch + subscription for real-time data": there is nothing here to subscribe TO. The + * writes happen in another process, and no in-process channel — a subject, an emitter, the sink + * above — can carry what it never sees. A subscription that cannot see the writes it is meant to + * relay is not one. + */ + async deliverPaymentUpdates(): Promise { + await this.deliverToWaitingCallers(); + await this.deliverToConnectedDevices(); + } + + private async deliverToWaitingCallers(): Promise { + const ids = this.paymentWaitMap.get(); + if (!ids.length) return; + + const payments = await this.paymentLinkPaymentRepo.find({ + where: { id: In(ids) }, + relations: { link: true }, + }); + + for (const payment of payments) { + if (this.waitStates.get(payment.id) !== payment.waitState) this.resolveWaiters(payment); + } + } + + private async deliverToConnectedDevices(): Promise { + const devices = this.connectedDevices(); + const cutoff = this.deliveryCutoff(); + + this.pruneDeliveries(cutoff); + + if (!devices.length) return; + + // One cutoff for all of them, because it no longer depends on the connection: a payment is read + // until its own end has passed, whoever is listening and since when. The condition is the one + // the direct delivery in doSave runs under, expressed over stored columns: a payment out of + // `Pending`, or a `MULTIPLE`-mode payment that has counted a completed quote. + const deviceIds = devices.map((device) => device.id); + const where = [ + { deviceId: In(deviceIds), expiryDate: MoreThan(cutoff), status: Not(PaymentLinkPaymentStatus.PENDING) }, + { deviceId: In(deviceIds), expiryDate: MoreThan(cutoff), txCount: MoreThan(0) }, + ]; + + const payments = await this.paymentLinkPaymentRepo.find({ where, order: { expiryDate: 'ASC' } }); + + for (const payment of payments) this.deliverToDevice(payment); + } + + /** + * The oldest `expiryDate` the read still asks for: a payment's own end, plus the delay before + * `processExpiredPayments` acts on it, plus the grace above. The delay is READ rather than + * assumed — raising `PAYMENT_TIMEOUT_DELAY` past a hard-coded span would otherwise drop the + * expiry transition out of the read without changing a line here. + */ + private deliveryCutoff(): Date { + return Util.secondsBefore(Config.payment.timeoutDelay + DEVICE_DELIVERY_GRACE_SECONDS); + } + + /** + * Drops what the read can no longer return. An entry is kept while its payment is still asked + * for, NOT while its device is connected: a device that reconnects finds its record intact and is + * not told a second time about what it already heard. The record therefore holds exactly what + * the read can still produce, and a device whose entries have all gone leaves with them. + * + * That ties the size of this map to the READ, not to a span: `expiryDate` comes from the caller + * (`CreatePaymentLinkPaymentDto`, optional and validated only as a date) and falls back to the + * link's `paymentTimeout` when it is left out. For the payments the API dates itself that means + * roughly the timeout plus an hour; a caller that sets an expiry far ahead keeps its payments in + * the read — and here — until then. Nothing in this class caps that, and capping it would be a + * change to what a merchant may ask for rather than to this delivery. + */ + private pruneDeliveries(cutoff: Date): void { + for (const [deviceId, delivered] of this.deviceDeliveries) { + for (const [paymentId, entry] of delivered) { + if (!(entry.expiryDate > cutoff)) delivered.delete(paymentId); + } + if (!delivered.size) this.deviceDeliveries.delete(deviceId); + } + } + + /** + * Drops what this process believes it told a device, so the next tick tells it again. + * + * Called by the gateway on the two signals that a command may have failed AFTER the sink + * answered `true`, both of which are invisible here: `'error'`, which is how `ws` reports a send + * on a socket that closed mid-call (the state was open when it was read, and the call did not + * throw), and the ping sweep dropping a peer that stopped answering without closing anything — + * there a send simply went into a socket nobody was reading. + * + * Deliberately NOT called on `'close'`. A peer that completes the closing handshake was reading + * until it did, so an orderly close is not evidence that anything failed; forgetting there would + * repeat every command on every reconnect, which is what the record exists to prevent. + */ + forgetDeliveries(deviceId: string): void { + this.deviceDeliveries.delete(deviceId); + } + + /** What has been delivered to a device so far, empty for one nothing has been sent to yet. */ + private deliveriesFor(deviceId: string): DeviceDeliveries { + const delivered = this.deviceDeliveries.get(deviceId) ?? new Map(); + this.deviceDeliveries.set(deviceId, delivered); + + return delivered; + } + + private resolveWaiters(payment: PaymentLinkPayment): void { + this.waitStates.delete(payment.id); + this.paymentWaitMap.resolve(payment.id, payment); + } + + /** + * Idempotent by the state it delivers: a device is sent the same command for the same payment + * state once, whether this process wrote it or read it back. Both callers go through here. + */ + private deliverToDevice(payment: PaymentLinkPayment): void { + const device = payment.device; + if (!device) return; + + const connected = this.connectedDevices().find((d) => d.id === device.id); + if (!connected) return; + + // Per payment rather than one slot per device: the read above holds several payments of the + // same device at once, and a single slot would let two of them take turns evicting each other. + const delivered = this.deliveriesFor(connected.id); + if (delivered.get(payment.id)?.state === payment.waitState) return; + + // Recorded only once the command is out, and that ordering is the whole point of the sink's + // return value. Recording first was wrong in a way the comment here used to paper over: it + // claimed a failed send would go out again "on the next tick under a new connection", but the + // record is keyed by DEVICE, not by connection, and `pruneDeliveries` keeps it precisely so a + // reconnecting device is not told twice. A send that threw would therefore have been recorded + // as delivered and never retried — a silent loss on the one path that exists to prevent one. + // + // The cost of this order is a repeat when the command arrives but the process dies before the + // record is written. Repeating is what `waitState` makes harmless; losing is not. + // + // What no ordering fixes: this record lives in the process. A restart empties it, and a device + // still inside the read gets its current state again — the same harmless repeat. + if (!this.deviceSink(device)) return; + + delivered.set(payment.id, { state: payment.waitState, expiryDate: payment.expiryDate }); + } + + async handleBinanceWaiting(result: C2BWebhookResult): Promise { + const { qrContent, referId } = result.metadata; + + const lnurl = new URL(qrContent).searchParams.get('lightning'); + const uniqueId = LightningHelper.decodeLnurl(lnurl).split('/').at(-1); + const payment = await this.getPendingPaymentByUniqueId(uniqueId); + if (!payment) throw new NotFoundException('Payment not found'); + + const quote = await this.paymentQuoteService.createQuote(payment.link.defaultStandard, payment); + const transferAmount = JSON.parse(quote.transferAmounts).find((t) => t.method === Blockchain.BINANCE_PAY); + if (!transferAmount?.assets.length) throw new NotFoundException('Transfer amount not found'); + + const transferInfo: TransferInfo = { + asset: transferAmount.assets[0].asset, + amount: transferAmount.assets[0].amount, + method: Blockchain.BINANCE_PAY, + quoteUniqueId: quote.uniqueId, + referId, + }; + + await this.createActivationRequest(payment.uniqueId, transferInfo); + } + + async createPayment(paymentLink: PaymentLink, dto: CreatePaymentLinkPaymentDto): Promise { + if (paymentLink.status !== PaymentLinkStatus.ACTIVE) throw new BadRequestException('Payment link is not active'); + + const pendingPayment = paymentLink.payments.some((p) => p.status === PaymentLinkPaymentStatus.PENDING); + if (pendingPayment) + throw new ConflictException('There is already a pending payment for the specified payment link'); + + if (paymentLink.mode === PaymentLinkMode.SINGLE) { + const hasPreviousPayment = await this.paymentLinkPaymentRepo.existsBy({ + link: { uniqueId: paymentLink.uniqueId }, + }); + if (hasPreviousPayment) throw new ConflictException('Single payment link can only have one payment'); + } + + if (dto.externalId) { + const exists = await this.paymentLinkPaymentRepo.existsBy({ + externalId: dto.externalId, + link: { id: paymentLink.id }, + }); + if (exists) throw new ConflictException('Payment already exists'); + } + + if (isSellRoute(paymentLink.route) && dto.currency && dto.currency !== paymentLink.route.fiat.name) + throw new BadRequestException('Payment currency mismatch'); + + const currency = isSellRoute(paymentLink.route) + ? paymentLink.route.fiat + : await this.fiatService.getFiatByName(dto.currency ?? 'CHF'); + + const payment = this.paymentLinkPaymentRepo.create({ + amount: dto.amount, + externalId: dto.externalId, + note: dto.note, + expiryDate: dto.expiryDate ?? Util.secondsAfter(paymentLink.configObj.paymentTimeout), + mode: dto.mode ?? PaymentLinkPaymentMode.SINGLE, + currency, + uniqueId: Util.createUniqueId(Config.prefixes.paymentLinkPaymentUidPrefix), + status: PaymentLinkPaymentStatus.PENDING, + link: paymentLink, + }); + + const savedPayment = await this.doSave(payment, false); + + // auto confirm (DEV only) + if (Config.environment !== Environment.PRD && paymentLink.configObj.autoConfirmSecs != null) { + setTimeout(async () => { + if (payment.amount === 0.01) { + payment.cancel(); + } else { + payment.complete(); + } + await this.doSave(payment, true); + }, paymentLink.configObj.autoConfirmSecs * 1000); + } + + // expiry timers + // + // These stay in the process that served the request, although processExpiredPayments is a + // worker job. Both therefore race for the same payment, and neither the lock nor the lease + // spans them; what settles it is that the transition itself is atomic, see + // takePendingTransition. What the timers buy is what they bought before — a caller waiting on + // this process is released at the timeout rather than at the next tick of a job elsewhere. + const scanTimeout = paymentLink.configObj.scanTimeout; + if (scanTimeout) { + setTimeout(() => this.expirePaymentIfPending(payment.id, true), scanTimeout * 1000); + } + + const paymentExpiry = Util.secondsAfter(Config.payment.timeoutDelay, payment.expiryDate); + if (Util.minutesDiff(new Date(), paymentExpiry) <= 60) { + const paymentTimeout = paymentExpiry.getTime() - new Date().getTime(); + setTimeout(() => this.expirePaymentIfPending(payment.id, false), paymentTimeout); + } + + return savedPayment; + } + + private async expirePaymentIfPending(id: number, ignoreWithQuote: boolean): Promise { + const pendingPayment = await this.paymentLinkPaymentRepo.findOne({ + where: { + id, + status: PaymentLinkPaymentStatus.PENDING, + quotes: { id: ignoreWithQuote ? IsNull() : undefined }, + }, + relations: { link: true }, + }); + + if (pendingPayment) await this.expirePayment(pendingPayment); + } + + async confirmPayment(payment: PaymentLinkPayment): Promise { + if (payment.status !== PaymentLinkPaymentStatus.COMPLETED) + throw new BadRequestException('Payment is not completed'); + + await this.paymentLinkPaymentRepo.update(payment.id, { isConfirmed: true }); + } + + async cancelByLink(paymentLink: PaymentLink): Promise { + const pendingPayment = paymentLink.payments.find((p) => p.status === PaymentLinkPaymentStatus.PENDING); + if (!pendingPayment) throw new NotFoundException('No pending payment found'); + + pendingPayment.link = paymentLink; + + await this.cancelByPayment(pendingPayment); + + return paymentLink; + } + + async cancelByPayment(payment: PaymentLinkPayment): Promise { + // Both callers reach here from a payment they read as `Pending`, and the worker can expire + // that same payment in between. Whoever the transition lets through sends the webhook. + const taken = await this.takePendingTransition(payment, PaymentLinkPaymentStatus.CANCELLED, (manager) => + this.cancelQuotesForPayment(payment.id, manager), + ); + if (!taken) return; + + await this.doSave(payment.cancel(), true); + } + + async deletePayment(payment: PaymentLinkPayment): Promise { + if (payment.status === PaymentLinkPaymentStatus.COMPLETED) + throw new BadRequestException('PaymentLinkPayment is already completed, cannot be deleted'); + + for (const quote of payment.quotes) { + await this.paymentQuoteService.deleteQuote(quote); + } + + for (const activation of payment.activations) { + await this.paymentActivationService.deleteActivation(activation); + } + + await this.paymentLinkPaymentRepo.delete(payment.id); + } + + /** The database effects of leaving `Pending`, run on the manager of the transition's transaction. */ + private async cancelQuotesForPayment(paymentId: number, manager: EntityManager): Promise { + await this.paymentQuoteService.cancelAllForPayment(paymentId, manager); + await this.paymentActivationService.closeAllForPayment(paymentId, manager); + } + + // --- HANDLE CALLBACKS --- // + async createActivationRequest( + uniqueId: string, + transferInfo: TransferInfo, + ): Promise { + const pendingPayment = await this.getPendingPaymentByUniqueId(uniqueId); + if (!pendingPayment) throw new NotFoundException(`Pending payment not found by id ${uniqueId}`); + + const activation = await this.paymentActivationService.doCreateRequest(pendingPayment, transferInfo); + return PaymentRequestMapper.toPaymentRequest(activation); + } + + async handleHexPayment(uniqueId: string, transferInfo: TransferInfo): Promise { + const pendingPayment = await this.getPendingPaymentByUniqueId(uniqueId); + if (!pendingPayment) throw new NotFoundException(`Pending payment not found by id ${uniqueId}`); + + const quote = await this.paymentQuoteService.executeHexPayment(transferInfo); + await this.handleQuoteChange(pendingPayment, quote); + + if (quote.status === PaymentQuoteStatus.TX_FAILED) + throw new BadRequestException(`Failed to handle hex payment ${uniqueId}: ${quote.errorMessage}`); + + return { txId: quote.txId }; + } + + // --- HANDLE INPUTS --- // + async getPaymentQuoteByFailedCryptoInput(cryptoInput: CryptoInput): Promise { + const quote = await this.paymentQuoteService.getQuoteByTxId(cryptoInput.address.blockchain, cryptoInput.inTxId, [ + PaymentQuoteStatus.TX_MEMPOOL, + PaymentQuoteStatus.TX_BLOCKCHAIN, + PaymentQuoteStatus.TX_COMPLETED, + ]); + if (!quote) return null; + + if (quote.status === PaymentQuoteStatus.TX_MEMPOOL) { + await this.handleBlockchainConfirmed(quote, cryptoInput); + } + + return quote; + } + + async getPaymentQuoteByCryptoInput(cryptoInput: CryptoInput): Promise { + const quote = await this.getQuoteForInput(cryptoInput); + if (!quote) throw new Error(`No matching quote found`); + + await this.handleBlockchainConfirmed(quote, cryptoInput); + + return quote; + } + + private async handleBlockchainConfirmed(quote: PaymentQuote, cryptoInput: CryptoInput): Promise { + await this.paymentQuoteService.saveBlockchainConfirmed(quote, cryptoInput.address.blockchain, cryptoInput.inTxId); + + const payment = await this.paymentLinkPaymentRepo.findOne({ + where: { id: quote.payment.id }, + relations: { link: { route: { user: { userData: { organization: true } } } } }, + }); + + await this.handleQuoteChange(payment, quote); + } + + private async getQuoteForInput(cryptoInput: CryptoInput): Promise { + const quote = [Blockchain.LIGHTNING, Blockchain.BINANCE_PAY, Blockchain.KUCOIN_PAY].includes( + cryptoInput.address.blockchain, + ) + ? await this.getQuoteByActivation(cryptoInput.address.blockchain, cryptoInput.inTxId) + : await this.getQuoteByTx(cryptoInput.address.blockchain, cryptoInput.inTxId); + + if (quote) return quote; + + return this.paymentQuoteService.getQuoteByAsset(cryptoInput.asset, cryptoInput.amount); + } + + private async getQuoteByActivation(txBlockchain: Blockchain, txId: string): Promise { + const activation = await this.paymentActivationService.getActivationByTxId(txId); + if (!activation) return null; + + const quote = activation.quote; + if (quote && !quote.txId) await this.paymentQuoteService.saveTransaction(quote, txBlockchain, txId); + + return quote; + } + + private async getQuoteByTx(txBlockchain: Blockchain, txId: string): Promise { + return this.paymentQuoteService.getQuoteByTxId(txBlockchain, txId, [ + PaymentQuoteStatus.TX_RECEIVED, + PaymentQuoteStatus.TX_MEMPOOL, + PaymentQuoteStatus.TX_BLOCKCHAIN, + ]); + } + + private async handleQuoteChange(payment: PaymentLinkPayment, quote: PaymentQuote): Promise { + // Closing the activations of a final quote happens on every path through here, but on ONE of + // them it has to travel with the transition rather than run before it — hence the closure. + // Given a manager it runs inside that transaction; without one it stands alone, as before. + const closeActivations = async (manager?: EntityManager): Promise => { + if (!PaymentQuoteFinalStates.includes(quote.status)) return; + + if (payment.mode === PaymentLinkPaymentMode.SINGLE) { + await this.paymentActivationService.closeAllForPayment(payment.id, manager); + } else { + await this.paymentActivationService.closeAllForQuote(quote.id); + } + }; + + if (payment.status !== PaymentLinkPaymentStatus.PENDING) return closeActivations(); + + // update payment status + const { minCompletionStatus } = payment.link.configObj; + + const isPaymentComplete = + PaymentQuoteTxStates.indexOf(quote.status) >= PaymentQuoteTxStates.indexOf(minCompletionStatus); + if (!isPaymentComplete) return closeActivations(); + + const txCount = await this.paymentQuoteService.getCompletedQuoteCount(payment, minCompletionStatus); + payment.txCount = txCount; + + // The status read above is the same read-then-write as in expirePayment, and this one is + // reached from request paths as well as from checkTxConfirmations. A `MULTIPLE` payment + // stays `Pending` and has no transition to take: it only counts a quote. + if (payment.mode === PaymentLinkPaymentMode.SINGLE) { + const taken = await this.takePendingTransition( + payment, + PaymentLinkPaymentStatus.COMPLETED, + // Both effects belong to the transition. The count, because `doSave` below is the only + // other thing that writes it and no job looks at a completed payment again — a count left + // behind by a caller that stopped between the two would stay wrong. The activations, + // because closing them BEFORE the status moves is the half-state nothing can repair: the + // quote is already final, so `checkTxConfirmations` does not come back to it, while + // `processExpiredPayments` only ever asks for `Pending` and would expire a payment whose + // activations are long closed. + async (manager) => { + await manager.update(PaymentLinkPayment, payment.id, { txCount }); + await closeActivations(manager); + }, + ); + if (!taken) return; + + payment.complete(); + } else { + await closeActivations(); + } + + await this.doSave(payment, true); + } + + private async doSave(payment: PaymentLinkPayment, isPaymentDone: boolean): Promise { + const savedPayment = await this.paymentLinkPaymentRepo.save(payment); + + if (savedPayment.link.webhookUrl) await this.sendWebhook(savedPayment); + + // Delivers to this process directly, which is the whole latency budget when the writing job + // and the waiting caller share a process. Whoever waits elsewhere is served by + // deliverPaymentUpdates, which reads the row this save just wrote. + if (isPaymentDone) { + this.resolveWaiters(savedPayment); + this.deliverToDevice(savedPayment); + } + + return savedPayment; + } + + private async sendWebhook(payment: PaymentLinkPayment): Promise { + const paymentForWebhook = await this.paymentLinkPaymentRepo.findOne({ + where: { uniqueId: payment.uniqueId }, + relations: { + link: { route: { user: { userData: { organization: true } } } }, + }, + }); + + const paymentLink = paymentForWebhook.link; + paymentLink.payments = [paymentForWebhook]; + + await this.paymentWebhookService.sendWebhook(paymentLink); + } +} diff --git a/src/subdomains/core/payment-link/services/payment-quote.service.ts b/src/subdomains/core/payment-link/services/payment-quote.service.ts index aa1577e99d..6fc48d9792 100644 --- a/src/subdomains/core/payment-link/services/payment-quote.service.ts +++ b/src/subdomains/core/payment-link/services/payment-quote.service.ts @@ -20,7 +20,7 @@ import { C2BPaymentLinkService } from 'src/subdomains/core/payment-link/services import { PaymentBalanceService } from 'src/subdomains/core/payment-link/services/payment-balance.service'; import { PaymentLinkFeeService } from 'src/subdomains/core/payment-link/services/payment-link-fee.service'; import { PriceValidity, PricingService } from 'src/subdomains/supporting/pricing/services/pricing.service'; -import { Equal, In, LessThan } from 'typeorm'; +import { EntityManager, Equal, In, LessThan, Repository } from 'typeorm'; import { TransferAmount, TransferAmountAsset, TransferInfo } from '../dto/payment-link.dto'; import { PaymentLinkPayment } from '../entities/payment-link-payment.entity'; import { PaymentLink } from '../entities/payment-link.entity'; @@ -173,13 +173,20 @@ export class PaymentQuoteService { }); } - async cancelAllForPayment(paymentId: number): Promise { - const actualQuotes = await this.paymentQuoteRepo.find({ + /** + * `manager` runs the cancellations in the caller's transaction. The caller is the payment leaving + * `Pending` (see `PaymentLinkPaymentService.takePendingTransition`), and these quotes have to + * leave `Actual` with it or not at all: a payment out of `Pending` is not looked at again. + */ + async cancelAllForPayment(paymentId: number, manager?: EntityManager): Promise { + const repo: Repository = manager?.getRepository(PaymentQuote) ?? this.paymentQuoteRepo; + + const actualQuotes = await repo.find({ where: { payment: { id: paymentId }, status: PaymentQuoteStatus.ACTUAL }, }); for (const actualQuote of actualQuotes) { - await this.paymentQuoteRepo.save(actualQuote.cancel()); + await repo.save(actualQuote.cancel()); } } diff --git a/src/subdomains/core/referral/process/ref.service.ts b/src/subdomains/core/referral/process/ref.service.ts index c952dbe942..61aeea4a76 100644 --- a/src/subdomains/core/referral/process/ref.service.ts +++ b/src/subdomains/core/referral/process/ref.service.ts @@ -1,7 +1,8 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { IsNull, LessThan } from 'typeorm'; import { Ref } from './ref.entity'; @@ -15,7 +16,7 @@ export class RefService { constructor(private readonly repo: RefRepository) {} - @DfxCron(CronExpression.EVERY_HOUR, { timeout: 7200 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.REF_CLEANUP, timeout: 7200 }) async checkRefs(): Promise { const expirationDate = Util.daysBefore(this.refExpirationDays); diff --git a/src/subdomains/core/referral/reward/services/ref-reward-job.service.ts b/src/subdomains/core/referral/reward/services/ref-reward-job.service.ts index 573a547e8f..033b95ff80 100644 --- a/src/subdomains/core/referral/reward/services/ref-reward-job.service.ts +++ b/src/subdomains/core/referral/reward/services/ref-reward-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { RefRewardDexService } from './ref-reward-dex.service'; import { RefRewardNotificationService } from './ref-reward-notification.service'; import { RefRewardOutService } from './ref-reward-out.service'; @@ -16,12 +16,12 @@ export class RefRewardJobService { private readonly refRewardService: RefRewardService, ) {} - @DfxCron(CronExpression.EVERY_DAY_AT_6AM, { process: Process.REF_PAYOUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_DAY_AT_6AM, { scope: CronScope.WORKER, process: Process.REF_PAYOUT, timeout: 1800 }) async createPendingRefRewards() { await this.refRewardService.createPendingRefRewards(); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.REF_PAYOUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.REF_PAYOUT, timeout: 1800 }) async processPendingRefRewards() { await this.refRewardDexService.secureLiquidity(); await this.refRewardOutService.checkPaidTransaction(); diff --git a/src/subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts b/src/subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts index f788e47664..175725084b 100644 --- a/src/subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts +++ b/src/subdomains/core/sell-crypto/process/services/buy-fiat-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { BuyFiatPreparationService } from './buy-fiat-preparation.service'; import { BuyFiatRegistrationService } from './buy-fiat-registration.service'; @@ -12,7 +12,7 @@ export class BuyFiatJobService { private readonly buyFiatPreparationService: BuyFiatPreparationService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BUY_FIAT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.BUY_FIAT, timeout: 1800 }) async checkCryptoPayIn() { await this.buyFiatRegistrationService.registerSellPayIn(); await this.buyFiatRegistrationService.syncReturnTxId(); @@ -26,7 +26,7 @@ export class BuyFiatJobService { await this.buyFiatPreparationService.chargebackTx(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BUY_FIAT, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.BUY_FIAT, timeout: 7200 }) async addFiatOutputs(): Promise { await this.buyFiatPreparationService.addFiatOutputs(); } diff --git a/src/subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts b/src/subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts index 8f110b2f79..add8170b9b 100644 --- a/src/subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts +++ b/src/subdomains/core/sell-crypto/process/services/buy-fiat-notification.service.ts @@ -2,7 +2,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { AmlReason, AmlReasonWithoutReason, KycAmlReasons } from 'src/subdomains/core/aml/enums/aml-reason.enum'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; @@ -30,7 +30,7 @@ export class BuyFiatNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BUY_FIAT_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.BUY_FIAT_MAIL, timeout: 1800 }) async sendNotificationMails(): Promise { await this.paymentCompleted(); await this.chargebackInitiated(); diff --git a/src/subdomains/core/sell-crypto/route/sell.service.ts b/src/subdomains/core/sell-crypto/route/sell.service.ts index 81117b5174..a389491662 100644 --- a/src/subdomains/core/sell-crypto/route/sell.service.ts +++ b/src/subdomains/core/sell-crypto/route/sell.service.ts @@ -17,7 +17,7 @@ import { AssetService } from 'src/shared/models/asset/asset.service'; import { AssetDtoMapper } from 'src/shared/models/asset/dto/asset-dto.mapper'; import { FiatDtoMapper } from 'src/shared/models/fiat/dto/fiat-dto.mapper'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CreateSellDto } from 'src/subdomains/core/sell-crypto/route/dto/create-sell.dto'; import { UpdateSellDto } from 'src/subdomains/core/sell-crypto/route/dto/update-sell.dto'; @@ -223,12 +223,12 @@ export class SellService { } // --- VOLUMES --- // - @DfxCron(CronExpression.EVERY_YEAR) + @DfxCron(CronExpression.EVERY_YEAR, { scope: CronScope.WORKER }) async resetAnnualVolumes(): Promise { await this.sellRepo.update({ annualVolume: Not(0) }, { annualVolume: 0 }); } - @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT) + @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT, { scope: CronScope.WORKER }) async resetMonthlyVolumes(): Promise { await this.sellRepo.update({ monthlyVolume: Not(0) }, { monthlyVolume: 0 }); } diff --git a/src/subdomains/core/statistic/__tests__/statistic.service.spec.ts b/src/subdomains/core/statistic/__tests__/statistic.service.spec.ts index 02efff7a79..5afe171716 100644 --- a/src/subdomains/core/statistic/__tests__/statistic.service.spec.ts +++ b/src/subdomains/core/statistic/__tests__/statistic.service.spec.ts @@ -1,10 +1,12 @@ import { createMock } from '@golevelup/ts-jest'; import { Test, TestingModule } from '@nestjs/testing'; +import { ConfigService, GetConfig } from 'src/config/config'; import { Setting } from 'src/shared/models/setting/setting.entity'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { BuyService } from 'src/subdomains/core/buy-crypto/routes/buy/buy.service'; import { SellService } from 'src/subdomains/core/sell-crypto/route/sell.service'; import { StatisticService } from 'src/subdomains/core/statistic/statistic.service'; +import * as ProcessService from 'src/shared/services/process.service'; import { UserService } from 'src/subdomains/generic/user/models/user/user.service'; describe('StatisticService', () => { @@ -52,4 +54,159 @@ describe('StatisticService', () => { await expect(service.getStatus()).resolves.toEqual({}); }); }); + + /** + * `doUpdate` is scoped `api` AND leased, so it runs in ONE api process per tick. Everything the + * request path reads therefore has to be fillable from the request path — CONTRIBUTING: "A cron + * job may refresh it, but must not be the only thing filling it." + */ + describe('getAll', () => { + it('fills the statistic when there is none, instead of answering with nothing', async () => { + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + await service.getAll(); + + expect(update).toHaveBeenCalledTimes(1); + }); + + it('serves what is on hand without asking again', async () => { + service['statistic'] = { at: new Date(), data: { totalVolume: { buy: 1, sell: 2 } } as never }; + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + await expect(service.getAll()).resolves.toEqual({ totalVolume: { buy: 1, sell: 2 } }); + expect(update).not.toHaveBeenCalled(); + }); + + it('refreshes what has aged past the job it complements', async () => { + // A second api process — a blue-green window, a `--scale` — loses the lease on every tick + // and would otherwise serve what it read at boot for the rest of its life. + service['statistic'] = { at: new Date(Date.now() - 61 * 60 * 1000), data: {} as never }; + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + await service.getAll(); + + expect(update).toHaveBeenCalledTimes(1); + }); + + it('starts ONE refresh for concurrent readers', async () => { + // The field is written when a refresh resolves, so without the in-flight promise every + // request arriving until then starts its own — and one refresh is four aggregations over + // the whole volume history. + let finish: () => void; + const update = jest + .spyOn(service, 'doUpdate') + .mockImplementation(() => new Promise((resolve) => (finish = resolve))); + + const calls = [service.getAll(), service.getAll(), service.getAll()]; + await new Promise((resolve) => setImmediate(resolve)); + finish(); + await Promise.all(calls); + + expect(update).toHaveBeenCalledTimes(1); + }); + + it('answers with what it has when the refresh fails', async () => { + // The endpoint is public and read-only: an aggregation that cannot run right now says + // nothing about the numbers already here. + service['statistic'] = { at: new Date(Date.now() - 61 * 60 * 1000), data: { old: true } as never }; + jest.spyOn(service, 'doUpdate').mockRejectedValue(new Error('aggregation failed')); + jest.spyOn(service['logger'], 'error').mockImplementation(); + + await expect(service.getAll()).resolves.toEqual({ old: true }); + }); + + it('lets the next reader retry after a failed refresh', async () => { + const update = jest + .spyOn(service, 'doUpdate') + .mockRejectedValueOnce(new Error('aggregation failed')) + .mockResolvedValue(undefined); + jest.spyOn(service['logger'], 'error').mockImplementation(); + + await service.getAll(); + await service.getAll(); + + expect(update).toHaveBeenCalledTimes(2); + }); + }); + + /** + * The start-up fill runs outside the scheduler, so none of the conditions the scheduler applies + * to `doUpdate` reached it: not the scope, not the process flag, not any error handling. Each + * test below is one of those. + */ + describe('start-up fill', () => { + const originalRole = process.env.CRON_ROLE; + + function withRole(role: string): void { + process.env.CRON_ROLE = role; + new ConfigService(GetConfig()); + } + + beforeEach(() => { + jest.spyOn(ProcessService, 'DisabledProcess').mockReturnValue(false); + }); + + afterEach(() => { + jest.restoreAllMocks(); + + if (originalRole == null) delete process.env.CRON_ROLE; + else process.env.CRON_ROLE = originalRole; + + new ConfigService(GetConfig()); + }); + + it('does not run in the worker process', () => { + // The job is scoped `api`: a request path is the only reader of the field it writes. Run + // here regardless, the worker spent the aggregation queries once per boot on a value no + // request in that process can read — and outside the lease, so nothing reported it. + withRole('worker'); + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + service.onModuleInit(); + + expect(update).not.toHaveBeenCalled(); + }); + + it.each(['api', 'all'])('runs in the %s process', (role) => { + // The counterpart: where the job belongs, getAll would answer with undefined until the + // first scheduled run an hour later. + withRole(role); + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + service.onModuleInit(); + + expect(update).toHaveBeenCalledTimes(1); + }); + + it('stays off when the process flag is off', () => { + // Switching a job off has to switch it off, not leave one run per deployment behind. + jest.spyOn(ProcessService, 'DisabledProcess').mockReturnValue(true); + withRole('api'); + const update = jest.spyOn(service, 'doUpdate').mockResolvedValue(undefined); + + service.onModuleInit(); + + expect(update).not.toHaveBeenCalled(); + }); + + it('reports a failed fill instead of leaving an unhandled rejection', async () => { + // `void this.doUpdate()` without a catch turns a failing query at boot into an unhandled + // rejection. + withRole('api'); + const failure = new Error('aggregation failed'); + jest.spyOn(service, 'doUpdate').mockRejectedValue(failure); + const error = jest.spyOn(service['logger'], 'error').mockImplementation(); + + const unhandled = jest.fn(); + process.on('unhandledRejection', unhandled); + + service.onModuleInit(); + await new Promise((resolve) => setImmediate(resolve)); + + process.off('unhandledRejection', unhandled); + + expect(unhandled).not.toHaveBeenCalled(); + expect(error).toHaveBeenCalledWith('Failed to fill the statistic at start-up:', failure); + }); + }); }); diff --git a/src/subdomains/core/statistic/statistic.service.ts b/src/subdomains/core/statistic/statistic.service.ts index 889efd607e..6628eb13a6 100644 --- a/src/subdomains/core/statistic/statistic.service.ts +++ b/src/subdomains/core/statistic/statistic.service.ts @@ -1,9 +1,10 @@ import { Injectable, OnModuleInit } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; -import { Config } from 'src/config/config'; +import { Config, CronRole } from 'src/config/config'; import { SettingService } from 'src/shared/models/setting/setting.service'; -import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { DfxLogger } from 'src/shared/services/dfx-logger'; +import { DisabledProcess, Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { SellService } from 'src/subdomains/core/sell-crypto/route/sell.service'; import { UserService } from 'src/subdomains/generic/user/models/user/user.service'; @@ -12,7 +13,21 @@ import { SettingStatus, StatisticDto } from './dto/statistic.dto'; @Injectable() export class StatisticService implements OnModuleInit { - private statistic: StatisticDto; + private readonly logger = new DfxLogger(StatisticService); + + /** How long a filled statistic is served before a reader refreshes it; matches the job's own hour. */ + private static readonly MAX_AGE_SECONDS = 60 * 60; + + private statistic?: { at: Date; data: StatisticDto }; + + /** + * The refresh currently in flight, so concurrent readers of a cold or stale statistic share one. + * + * The field is only written when a refresh RESOLVES, so without this every request arriving + * until then would start its own — and one refresh is four aggregations over the whole volume + * history, a job that declares `timeout: 7200` for a reason. + */ + private refreshing?: Promise; constructor( private readonly buyService: BuyService, @@ -22,12 +37,31 @@ export class StatisticService implements OnModuleInit { ) {} onModuleInit() { - void this.doUpdate(); + // Fills the statistic once at start-up instead of leaving getAll answering with undefined + // until the first scheduled run an hour later. The three conditions below are the ones the + // scheduler applies to the job itself, and this call bypasses the scheduler entirely. + // + // The role first: the job is scoped `api` because a request path is the only reader of the + // field it writes. Run here unconditionally, both processes execute it once at boot — outside + // the cross-process lease, so nothing notices — and the worker spends the aggregation queries + // on a value no request in that process can read. + if (Config.cronRole === CronRole.WORKER) return; + + // Then the flag: a job switched off through DISABLED_PROCESSES has to stay off, including at + // start-up. Otherwise switching it off still leaves one run per deployment. + if (DisabledProcess(Process.UPDATE_STATISTIC)) return; + + void this.doUpdate().catch((e) => + // Not rethrown: an unhandled rejection here takes the process down over a statistic, and the + // scheduled run retries within the hour. Logged rather than swallowed, so the empty response + // in the meantime has a reason on record. + this.logger.error('Failed to fill the statistic at start-up:', e), + ); } - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.UPDATE_STATISTIC, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.API, process: Process.UPDATE_STATISTIC, timeout: 7200 }) async doUpdate(): Promise { - this.statistic = { + const data: StatisticDto = { totalVolume: { buy: Util.round(await this.buyService.getTotalVolume(), Config.defaultVolumeDecimal), sell: Util.round(await this.sellService.getTotalVolume(), Config.defaultVolumeDecimal), @@ -38,6 +72,10 @@ export class StatisticService implements OnModuleInit { }, status: await this.getStatus(), }; + + // Assigned as a whole, once every field is in. Written field by field, a reader arriving + // mid-refresh would see a statistic whose volumes and status come from different moments. + this.statistic = { at: new Date(), data }; } async getStatus(): Promise { @@ -45,7 +83,30 @@ export class StatisticService implements OnModuleInit { return settings.reduce((prev, curr) => ({ ...prev, [curr.key.replace('Status', '')]: curr.value }), {}); } - getAll(): StatisticDto { - return this.statistic; + /** + * Serves the statistic, refreshing it here when what is on hand is missing or older than the + * job's own interval. + * + * CONTRIBUTING: "A cron job may refresh it, but must not be the only thing filling it." The job + * above refreshes; this is the load, and it is what the lease made necessary. Scoped `api` AND + * leased, the job runs in ONE api process per tick — a second one (a blue-green window, a + * `--scale`) would otherwise serve whatever it read at boot for the rest of its life, with no + * later tick ever reaching it. The `DisabledProcess` switch has the same effect on the process + * that holds the lease. + */ + async getAll(): Promise { + const fresh = this.statistic && Util.secondsDiff(this.statistic.at) <= StatisticService.MAX_AGE_SECONDS; + + if (!fresh) { + try { + await (this.refreshing ??= this.doUpdate().finally(() => (this.refreshing = undefined))); + } catch (e) { + // What is on hand beats failing the request: this endpoint is public and read-only, and + // an aggregation that cannot run right now says nothing about the numbers already here. + this.logger.error('Failed to refresh the statistic on demand:', e); + } + } + + return this.statistic?.data; } } diff --git a/src/subdomains/core/trading/services/trading-job.service.ts b/src/subdomains/core/trading/services/trading-job.service.ts index af6cb02102..4e895fb7a8 100644 --- a/src/subdomains/core/trading/services/trading-job.service.ts +++ b/src/subdomains/core/trading/services/trading-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { TradingOrderService } from './trading-order.service'; import { TradingRuleService } from './trading-rule.service'; @@ -14,19 +14,19 @@ export class TradingJobService { // --- RULES --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.TRADING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.TRADING, timeout: 1800 }) async processRules() { await this.ruleService.processRules(); } - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.TRADING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.TRADING, timeout: 1800 }) async reactivateRules(): Promise { await this.ruleService.reactivateRules(); } // --- ORDERS --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.TRADING, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.TRADING, timeout: 1800 }) async processOrders() { await this.orderService.processOrders(); } diff --git a/src/subdomains/generic/admin/admin.service.ts b/src/subdomains/generic/admin/admin.service.ts index 34db9b377f..c74b111635 100644 --- a/src/subdomains/generic/admin/admin.service.ts +++ b/src/subdomains/generic/admin/admin.service.ts @@ -10,7 +10,7 @@ import { EvmBlockchains } from 'src/integration/blockchain/shared/util/blockchai import { AssetService } from 'src/shared/models/asset/asset.service'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { LiquidityOrderContext } from 'src/subdomains/supporting/dex/entities/liquidity-order.entity'; import { ReserveLiquidityRequest } from 'src/subdomains/supporting/dex/interfaces'; import { DexService } from 'src/subdomains/supporting/dex/services/dex.service'; @@ -79,7 +79,7 @@ export class AdminService { } } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_OUT, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_OUT, timeout: 3600 }) async completeLiquidityOrders() { for (const context of Object.values(PayoutRequestContext)) { const lContext = context as unknown as LiquidityOrderContext; diff --git a/src/subdomains/generic/kyc/services/kyc-notification.service.ts b/src/subdomains/generic/kyc/services/kyc-notification.service.ts index a9803e30a0..014481e755 100644 --- a/src/subdomains/generic/kyc/services/kyc-notification.service.ts +++ b/src/subdomains/generic/kyc/services/kyc-notification.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailKey, MailTranslationKey } from 'src/subdomains/supporting/notification/factories/mail.factory'; @@ -26,7 +26,7 @@ export class KycNotificationService { private readonly webhookService: WebhookService, ) {} - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.KYC_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.KYC_MAIL, timeout: 1800 }) async sendNotificationMails(): Promise { await this.autoKycStepReminder(); } diff --git a/src/subdomains/generic/kyc/services/kyc.service.ts b/src/subdomains/generic/kyc/services/kyc.service.ts index 92de4fa10f..585bc094ed 100644 --- a/src/subdomains/generic/kyc/services/kyc.service.ts +++ b/src/subdomains/generic/kyc/services/kyc.service.ts @@ -20,7 +20,7 @@ import { IEntity, UpdateResult } from 'src/shared/models/entity'; import { LanguageService } from 'src/shared/models/language/language.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { QueueHandler } from 'src/shared/utils/queue-handler'; import { Util } from 'src/shared/utils/util'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; @@ -134,7 +134,7 @@ export class KycService { this.webhookQueue = new QueueHandler(); } - @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { process: Process.KYC }) + @DfxCron(CronExpression.EVERY_DAY_AT_4AM, { scope: CronScope.WORKER, process: Process.KYC }) async checkIdentSteps(): Promise { const expiredIdentSteps = await this.kycStepRepo.find({ where: { @@ -162,7 +162,7 @@ export class KycService { } } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.KYC }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.KYC }) async reviewKycSteps(): Promise { await this.reviewNationalityStep(); await this.reviewIdentSteps(); diff --git a/src/subdomains/generic/kyc/services/tfa.service.ts b/src/subdomains/generic/kyc/services/tfa.service.ts index 6870109209..c1ae401bbf 100644 --- a/src/subdomains/generic/kyc/services/tfa.service.ts +++ b/src/subdomains/generic/kyc/services/tfa.service.ts @@ -12,7 +12,7 @@ import { CronExpression } from '@nestjs/schedule'; import { generateSecret, verifyToken } from 'node-2fa'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { TfaLogRepository } from 'src/subdomains/generic/kyc/repositories/tfa-log.repository'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; @@ -50,7 +50,7 @@ export class TfaService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.TFA_CACHE }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.BOTH, process: Process.TFA_CACHE }) processCleanupSecretCache() { const now = new Date(); diff --git a/src/subdomains/generic/user/models/auth/auth-lnurl.service.ts b/src/subdomains/generic/user/models/auth/auth-lnurl.service.ts index 4eb286e26b..05ac59373e 100644 --- a/src/subdomains/generic/user/models/auth/auth-lnurl.service.ts +++ b/src/subdomains/generic/user/models/auth/auth-lnurl.service.ts @@ -6,7 +6,7 @@ import { Config } from 'src/config/config'; import { LightningHelper } from 'src/integration/lightning/lightning-helper'; import { IpLogService } from 'src/shared/models/ip-log/ip-log.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { AuthService } from 'src/subdomains/generic/user/models/auth/auth.service'; import { @@ -36,7 +36,7 @@ export class AuthLnUrlService { private readonly ipLogService: IpLogService, ) {} - @DfxCron(CronExpression.EVERY_30_SECONDS, { process: Process.LNURL_AUTH_CACHE }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.BOTH, process: Process.LNURL_AUTH_CACHE }) processCleanupAccessToken() { const before30SecTime = Util.secondsBefore(30).getTime(); @@ -47,7 +47,7 @@ export class AuthLnUrlService { keysToBeDeleted.forEach((k) => this.authCache.delete(k)); } - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.LNURL_AUTH_CACHE }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.BOTH, process: Process.LNURL_AUTH_CACHE }) processCleanupAuthCache() { const before5MinTime = Util.minutesBefore(5).getTime(); diff --git a/src/subdomains/generic/user/models/auth/auth.service.ts b/src/subdomains/generic/user/models/auth/auth.service.ts index 258aa075b0..2dba8df2d2 100644 --- a/src/subdomains/generic/user/models/auth/auth.service.ts +++ b/src/subdomains/generic/user/models/auth/auth.service.ts @@ -22,7 +22,7 @@ import { LanguageService } from 'src/shared/models/language/language.service'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { AsyncCache, CacheItemResetPeriod } from 'src/shared/utils/async-cache'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { RefService } from 'src/subdomains/core/referral/process/ref.service'; import { KycStepName } from 'src/subdomains/generic/kyc/enums/kyc-step-name.enum'; @@ -98,7 +98,7 @@ export class AuthService { @Inject(forwardRef(() => KycService)) private readonly kycService: KycService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.BOTH }) checkLists() { for (const [key, challenge] of this.challengeList.entries()) { if (!this.isChallengeValid(challenge)) { diff --git a/src/subdomains/generic/user/models/bank-data/bank-data.service.ts b/src/subdomains/generic/user/models/bank-data/bank-data.service.ts index 538362cc63..af1714620b 100644 --- a/src/subdomains/generic/user/models/bank-data/bank-data.service.ts +++ b/src/subdomains/generic/user/models/bank-data/bank-data.service.ts @@ -6,7 +6,7 @@ import { CountryService } from 'src/shared/models/country/country.service'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { KycStepName } from 'src/subdomains/generic/kyc/enums/kyc-step-name.enum'; import { ReviewStatus } from 'src/subdomains/generic/kyc/enums/review-status.enum'; @@ -46,7 +46,11 @@ export class BankDataService { private readonly kycAdminService: KycAdminService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BANK_DATA_VERIFICATION, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.BANK_DATA_VERIFICATION, + timeout: 1800, + }) async checkAndSetActive() { await this.checkUnverifiedBankDatas(); } diff --git a/src/subdomains/generic/user/models/organization/organization.service.ts b/src/subdomains/generic/user/models/organization/organization.service.ts index 97d4fa2d9c..0227d6fd0c 100644 --- a/src/subdomains/generic/user/models/organization/organization.service.ts +++ b/src/subdomains/generic/user/models/organization/organization.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { CountryService } from 'src/shared/models/country/country.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { In, IsNull } from 'typeorm'; import { AccountType } from '../user-data/account-type.enum'; import { UserDataRepository } from '../user-data/user-data.repository'; @@ -21,7 +21,7 @@ export class OrganizationService { private readonly userDataRepo: UserDataRepository, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.ORGANIZATION_SYNC, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.ORGANIZATION_SYNC, timeout: 1800 }) async syncOrganization() { const entities = await this.userDataRepo.findBy({ organization: { id: IsNull() }, diff --git a/src/subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts b/src/subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts index 7a50409395..23ce2fad16 100644 --- a/src/subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts +++ b/src/subdomains/generic/user/models/user-data/jwt-revocation-sync.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { SettingService } from 'src/shared/models/setting/setting.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { In } from 'typeorm'; import { RiskStatus, UserDataStatus } from './user-data.enum'; import { UserDataRepository } from './user-data.repository'; @@ -25,7 +25,14 @@ export class JwtRevocationSyncService { // Runs every minute: fast revocation of a blocked or compromised account is a security requirement that // warrants the security-revocation exception to the "prefer 15min" cron guideline. - @DfxCron(CronExpression.EVERY_MINUTE, { timeout: 1800 }) + // + // And deliberately WITHOUT a `process` flag, for the same reason and matching + // StaffKycClearanceService::syncStaffKycClearance, which states it there. A flag is a switch that + // turns the job off in the database, `DISABLED_PROCESSES='*'` turns off everything that has one, + // and switched off this job does not empty the auto denylist — it stops writing to it. Accounts + // blocked after that keep their live JWTs, and nothing reports it: the role heartbeat goes on + // saying `lease ok`. A switch whose use is silent does not belong on a revocation path. + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, timeout: 1800 }) async syncDeniedJwtAccounts(): Promise { const blockedAccounts = await this.userDataRepo.find({ select: { id: true }, diff --git a/src/subdomains/generic/user/models/user-data/user-data-job.service.ts b/src/subdomains/generic/user/models/user-data/user-data-job.service.ts index 4cfa08916a..5559d9c69f 100644 --- a/src/subdomains/generic/user/models/user-data/user-data-job.service.ts +++ b/src/subdomains/generic/user/models/user-data/user-data-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; import { FileType } from 'src/subdomains/generic/kyc/dto/kyc-file.dto'; import { KycStepName } from 'src/subdomains/generic/kyc/enums/kyc-step-name.enum'; @@ -15,7 +15,7 @@ import { UserDataRepository } from './user-data.repository'; export class UserDataJobService { constructor(private readonly userDataRepo: UserDataRepository) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.USER_DATA, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.USER_DATA, timeout: 1800 }) async fillUserData() { await this.bankTxVerification(); await this.setAccountOpener(); diff --git a/src/subdomains/generic/user/models/user-data/user-data-notification.service.ts b/src/subdomains/generic/user/models/user-data/user-data-notification.service.ts index d02f72499c..017b5ce23c 100644 --- a/src/subdomains/generic/user/models/user-data/user-data-notification.service.ts +++ b/src/subdomains/generic/user/models/user-data/user-data-notification.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailKey, MailTranslationKey } from 'src/subdomains/supporting/notification/factories/mail.factory'; @@ -20,7 +20,7 @@ export class UserDataNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.BLACK_SQUAD_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.BLACK_SQUAD_MAIL, timeout: 1800 }) async sendNotificationMails(): Promise { await this.blackSquadInvitation(); } diff --git a/src/subdomains/generic/user/models/user-data/user-data.service.ts b/src/subdomains/generic/user/models/user-data/user-data.service.ts index 0729bfa974..32523dcbd1 100644 --- a/src/subdomains/generic/user/models/user-data/user-data.service.ts +++ b/src/subdomains/generic/user/models/user-data/user-data.service.ts @@ -23,7 +23,7 @@ import { SettingService } from 'src/shared/models/setting/setting.service'; import { RepositoryFactory } from 'src/shared/repositories/repository.factory'; import { ApiKeyService } from 'src/shared/services/api-key.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { AmountType, Util } from 'src/shared/utils/util'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; import { CustodyService } from 'src/subdomains/core/custody/services/custody.service'; @@ -833,7 +833,7 @@ export class UserDataService { return this.doUpdateUserMail(userData, cacheEntry.mail); } - @DfxCron(CronExpression.EVERY_MINUTE) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.BOTH }) processCleanupMailSecretCache(): void { const now = new Date(); @@ -1191,7 +1191,7 @@ export class UserDataService { } // --- VOLUMES --- // - @DfxCron(CronExpression.EVERY_YEAR) + @DfxCron(CronExpression.EVERY_YEAR, { scope: CronScope.WORKER }) async resetAnnualVolumes(): Promise { await this.userDataRepo.update( [{ annualBuyVolume: Not(0) }, { annualSellVolume: Not(0) }, { annualCryptoVolume: Not(0) }], @@ -1199,7 +1199,7 @@ export class UserDataService { ); } - @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT) + @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT, { scope: CronScope.WORKER }) async resetMonthlyVolumes(): Promise { await this.userDataRepo.update( [{ monthlyBuyVolume: Not(0) }, { monthlySellVolume: Not(0) }, { monthlyCryptoVolume: Not(0) }], diff --git a/src/subdomains/generic/user/models/user/staff-kyc-clearance.service.ts b/src/subdomains/generic/user/models/user/staff-kyc-clearance.service.ts index 6aec42ed42..35f537c445 100644 --- a/src/subdomains/generic/user/models/user/staff-kyc-clearance.service.ts +++ b/src/subdomains/generic/user/models/user/staff-kyc-clearance.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { rolesSatisfying } from 'src/shared/auth/role.guard'; import { KycGatedRoles } from 'src/shared/auth/user-role.enum'; import { SettingService } from 'src/shared/models/setting/setting.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { In, Raw } from 'typeorm'; import { UserRepository } from './user.repository'; @@ -51,7 +51,12 @@ export class StaffKycClearanceService { // Every minute, matching JwtRevocationSyncService: revoking elevated access promptly is a security // requirement and warrants the same exception to the "prefer 15min" cron guideline. - @DfxCron(CronExpression.EVERY_MINUTE, { timeout: 1800 }) + // + // And deliberately WITHOUT a `process` flag, also matching that service: switched off, this job + // does not empty the clearance list, it stops maintaining it — staff blocked afterwards keep + // their elevated access, and nothing reports the state. A switch whose use is silent does not + // belong on a revocation path. + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, timeout: 1800 }) async syncStaffKycClearance(): Promise { const staffUsers = await this.userRepo.find({ select: { id: true, userData: { id: true } }, diff --git a/src/subdomains/generic/user/models/user/user-job.service.ts b/src/subdomains/generic/user/models/user/user-job.service.ts index a0b91e43ea..ba82bed7d0 100644 --- a/src/subdomains/generic/user/models/user/user-job.service.ts +++ b/src/subdomains/generic/user/models/user/user-job.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { FileType } from 'src/subdomains/generic/kyc/dto/kyc-file.dto'; import { IsNull, Like, MoreThan } from 'typeorm'; import { UserRepository } from './user.repository'; @@ -10,7 +10,7 @@ import { UserRepository } from './user.repository'; export class UserJobService { constructor(private readonly userRepo: UserRepository) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.USER, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.USER, timeout: 1800 }) async fillUser() { await this.approveUser(); } diff --git a/src/subdomains/generic/user/models/user/user.service.ts b/src/subdomains/generic/user/models/user/user.service.ts index b5d4d3d72e..d6bba889a4 100644 --- a/src/subdomains/generic/user/models/user/user.service.ts +++ b/src/subdomains/generic/user/models/user/user.service.ts @@ -20,7 +20,7 @@ import { LanguageDtoMapper } from 'src/shared/models/language/dto/language-dto.m import { LanguageService } from 'src/shared/models/language/language.service'; import { ApiKeyService } from 'src/shared/services/api-key.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; import { HistoryFilter, HistoryFilterKey } from 'src/subdomains/core/history/dto/history-filter.dto'; @@ -541,7 +541,7 @@ export class UserService { } // --- VOLUMES --- // - @DfxCron(CronExpression.EVERY_YEAR) + @DfxCron(CronExpression.EVERY_YEAR, { scope: CronScope.WORKER }) async resetAnnualVolumes(): Promise { await this.userRepo.update( [{ annualBuyVolume: Not(0) }, { annualSellVolume: Not(0) }, { annualCryptoVolume: Not(0) }], @@ -549,7 +549,7 @@ export class UserService { ); } - @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT) + @DfxCron(CronExpression.EVERY_1ST_DAY_OF_MONTH_AT_MIDNIGHT, { scope: CronScope.WORKER }) async resetMonthlyVolumes(): Promise { await this.userRepo.update( [{ monthlyBuyVolume: Not(0) }, { monthlySellVolume: Not(0) }, { monthlyCryptoVolume: Not(0) }], diff --git a/src/subdomains/generic/user/services/webhook/webhook-notification.service.ts b/src/subdomains/generic/user/services/webhook/webhook-notification.service.ts index 2c0570d69a..01981a4911 100644 --- a/src/subdomains/generic/user/services/webhook/webhook-notification.service.ts +++ b/src/subdomains/generic/user/services/webhook/webhook-notification.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { HttpService } from 'src/shared/services/http.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; import { IsNull } from 'typeorm'; @@ -23,7 +23,7 @@ export class WebhookNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.WEBHOOK, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.WEBHOOK, timeout: 1800 }) async sendWebhooks() { await this.sendOpenWebhooks(); } diff --git a/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts b/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts index 253a7e4eaf..238f30e8e5 100644 --- a/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts +++ b/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return-notification.service.ts @@ -2,7 +2,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { IsNull, Not } from 'typeorm'; import { MailContext, MailType } from '../../notification/enums'; import { MailKey, MailTranslationKey } from '../../notification/factories/mail.factory'; @@ -18,7 +18,11 @@ export class BankTxReturnNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.BANK_TX_RETURN_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.BANK_TX_RETURN_MAIL, + timeout: 1800, + }) async sendBankTxReturnMail() { await this.chargebackInitiated(); } diff --git a/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts b/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts index 9a98488e3c..413785001e 100644 --- a/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts +++ b/src/subdomains/supporting/bank-tx/bank-tx-return/bank-tx-return.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BankTxRefund, RefundInternalDto } from 'src/subdomains/core/history/dto/refund-internal.dto'; import { TransactionUtilService } from 'src/subdomains/core/transaction/transaction-util.service'; @@ -35,7 +35,7 @@ export class BankTxReturnService { private readonly fiatService: FiatService, ) {} - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.BANK_TX_RETURN, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.BANK_TX_RETURN, timeout: 1800 }) async fillBankTxReturn() { await this.chargebackTx(); await this.setFiatAmounts(); diff --git a/src/subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts b/src/subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts index 2853cdb53f..1e05d35ed9 100644 --- a/src/subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts +++ b/src/subdomains/supporting/bank-tx/bank-tx/services/bank-tx.service.ts @@ -15,7 +15,7 @@ import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { AmountType, Util } from 'src/shared/utils/util'; import { BuyCryptoService } from 'src/subdomains/core/buy-crypto/process/services/buy-crypto.service'; import { BuyService } from 'src/subdomains/core/buy-crypto/routes/buy/buy.service'; @@ -132,7 +132,7 @@ export class BankTxService implements OnModuleInit { } // --- TRANSACTION HANDLING --- // - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 3600, process: Process.BANK_TX }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.WORKER, timeout: 3600, process: Process.BANK_TX }) async checkBankTx(): Promise { try { await this.checkTransactions(); @@ -148,7 +148,7 @@ export class BankTxService implements OnModuleInit { await this.fillBankTx(); } - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.BANK_TX }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.BANK_TX }) async enrichYapealTransactions(): Promise { const transactions = await this.bankTxRepo.find({ where: { familyCode: 'CCRD' }, // credit card => wrong data diff --git a/src/subdomains/supporting/bank/bank-account/bank-account.service.ts b/src/subdomains/supporting/bank/bank-account/bank-account.service.ts index 7a7cbce1bb..04b3487bbf 100644 --- a/src/subdomains/supporting/bank/bank-account/bank-account.service.ts +++ b/src/subdomains/supporting/bank/bank-account/bank-account.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { BankDetailsDto, IbanDetailsDto, IbanService } from 'src/integration/bank/services/iban.service'; import { CountryService } from 'src/shared/models/country/country.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { KycType } from 'src/subdomains/generic/user/models/user-data/user-data.enum'; import { Equal, IsNull, Like, Not } from 'typeorm'; import { BankAccount, BankAccountInfos } from './bank-account.entity'; @@ -35,7 +35,7 @@ export class BankAccountService { // --- INTERNAL METHODS --- // - @DfxCron(CronExpression.EVERY_WEEK, { process: Process.BANK_ACCOUNT, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_WEEK, { scope: CronScope.WORKER, process: Process.BANK_ACCOUNT, timeout: 3600 }) async checkFailedBankAccounts(): Promise { const failedBankAccounts = await this.bankAccountRepo.findBy({ returnCode: 256 }); for (const bankAccount of failedBankAccounts) { @@ -43,7 +43,7 @@ export class BankAccountService { } } - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.BANK_ACCOUNT, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.BANK_ACCOUNT, timeout: 3600 }) async reloadErrorBankAccounts(): Promise { const bankAccounts = await this.bankAccountRepo.findBy({ result: Like('Error:%') }); for (const bankAccount of bankAccounts) { @@ -51,7 +51,7 @@ export class BankAccountService { } } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.BANK_ACCOUNT, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.BANK_ACCOUNT, timeout: 3600 }) async reloadUncheckedBankAccounts(): Promise { const bankAccounts = await this.bankAccountRepo.findBy({ result: IsNull(), iban: Not(IsNull()) }); for (const bankAccount of bankAccounts) { diff --git a/src/subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts b/src/subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts index 84ee88b35e..4a3bb2e8e3 100644 --- a/src/subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts +++ b/src/subdomains/supporting/bank/virtual-iban/virtual-iban-frick-issuance-reconciliation.service.ts @@ -4,7 +4,7 @@ import { FrickVirtualIban, FrickVirtualIbanState } from 'src/integration/bank/dt import { FrickVirtualIbansFetchResult } from 'src/integration/bank/services/frick.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { IbanBankName } from 'src/subdomains/supporting/bank/bank/dto/bank.dto'; import { FrickVibanProvider } from 'src/subdomains/supporting/bank/virtual-iban/providers/frick-viban.provider'; import { VirtualIbanIssuanceEvent } from 'src/subdomains/supporting/bank/virtual-iban/virtual-iban-issuance-event.entity'; @@ -88,6 +88,7 @@ export class VirtualIbanFrickIssuanceReconciliationService { * and external cleanup targets exact vIBAN identities, making repeated work fail closed or idempotent. */ @DfxCron(CronExpression.EVERY_HOUR, { + scope: CronScope.WORKER, process: Process.VIRTUAL_IBAN_FRICK_ISSUANCE_RECONCILIATION, timeout: 1800, }) diff --git a/src/subdomains/supporting/dashboard/__tests__/dashboard-financial.service.spec.ts b/src/subdomains/supporting/dashboard/__tests__/dashboard-financial.service.spec.ts index 02b280f7b7..e9f9d74943 100644 --- a/src/subdomains/supporting/dashboard/__tests__/dashboard-financial.service.spec.ts +++ b/src/subdomains/supporting/dashboard/__tests__/dashboard-financial.service.spec.ts @@ -10,18 +10,27 @@ import { FinancialLogSummary } from '../../log/log.repository'; import { LogService } from '../../log/log.service'; import { DashboardFinancialService } from '../dashboard-financial.service'; import { LatestBalanceResponseDto } from '../dto/financial-log.dto'; +import { TestUtil } from 'src/shared/utils/test.util'; import { LatestBalanceStore } from '../latest-balance.store'; describe('DashboardFinancialService', () => { let service: DashboardFinancialService; let logService: LogService; let assetService: AssetService; + /** + * The real store, not a double: what is under test here is that a request fills it, so a double + * answering `get` with whatever it was told would test the opposite of the point. + */ let latestBalanceStore: LatestBalanceStore; + afterEach(() => { + jest.restoreAllMocks(); + }); + beforeEach(async () => { logService = createMock(); assetService = createMock(); - latestBalanceStore = createMock(); + latestBalanceStore = new LatestBalanceStore(); const module: TestingModule = await Test.createTestingModule({ providers: [ @@ -30,12 +39,38 @@ describe('DashboardFinancialService', () => { { provide: AssetService, useValue: assetService }, { provide: RefRewardService, useValue: createMock() }, { provide: LatestBalanceStore, useValue: latestBalanceStore }, + TestUtil.provideConfig(), ], }).compile(); service = module.get(DashboardFinancialService); }); + function logEntry(): Log { + return { + id: 1, + created: new Date('2026-07-14T12:00:00Z'), + message: JSON.stringify({ assets: {}, balancesByFinancialType: {} }), + } as Log; + } + + /** Mocks the log entry and the assets the aggregation resolves, then runs the refresh job. */ + async function refreshFrom( + timestamp: Date, + assetLog: AssetLog, + balancesByFinancialType: BalancesByFinancialType, + assets: Asset[], + ): Promise { + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue({ + id: 1, + created: timestamp, + message: JSON.stringify({ assets: assetLog, balancesByFinancialType }), + } as Log); + jest.spyOn(assetService, 'getAssetsById').mockResolvedValue(assets); + + await service.refreshLatestBalance(); + } + function mapEntry(changes: unknown) { const log = { created: new Date(), message: JSON.stringify({ changes }) } as Log; return (service as any).mapChangesLogToEntry(log); @@ -307,33 +342,69 @@ describe('DashboardFinancialService', () => { }); }); - describe('getLatestBalance (write-through store read)', () => { - it('returns undefined when the store is empty and never touches the database', async () => { - jest.spyOn(latestBalanceStore, 'get').mockReturnValue(undefined); - const getLatestFinancialLogSpy = jest.spyOn(logService, 'getLatestFinancialLog'); - const getAssetsByIdSpy = jest.spyOn(assetService, 'getAssetsById'); - - const result = await service.getLatestBalance(); + describe('getLatestBalance (cached read that loads itself)', () => { + it('returns undefined when the database holds no financial log at all', async () => { + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(undefined); - expect(result).toBeUndefined(); - expect(getLatestFinancialLogSpy).toHaveBeenCalledTimes(0); - expect(getAssetsByIdSpy).toHaveBeenCalledTimes(0); + await expect(service.getLatestBalance()).resolves.toBeUndefined(); }); - it('returns exactly the value held in the store (pure store read)', async () => { - const cached: LatestBalanceResponseDto = { + it('loads the aggregate itself when no job has filled the store in this process', async () => { + // The refresh job is leased: with several API processes it runs in one of them per tick, so + // a request served anywhere else has to be able to fill the store on its own. + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(logEntry()); + + await expect(service.getLatestBalance()).resolves.toEqual({ timestamp: new Date('2026-07-14T12:00:00Z'), - byType: [{ name: 'Crypto', plusBalanceChf: 100, minusBalanceChf: 0, netBalanceChf: 100 }], - byBlockchain: [{ name: 'Ethereum', plusBalanceChf: 100, minusBalanceChf: 0, netBalanceChf: 100 }], - }; - jest.spyOn(latestBalanceStore, 'get').mockReturnValue(cached); + byType: [], + byBlockchain: [], + }); + }); + + it('does not read the database again while the entry it holds is current', async () => { + const getLatestFinancialLogSpy = jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(logEntry()); - await expect(service.getLatestBalance()).resolves.toBe(cached); + await service.getLatestBalance(); + await service.getLatestBalance(); + + expect(getLatestFinancialLogSpy).toHaveBeenCalledTimes(1); + }); + + it('runs one load for the requests that arrive together on an empty store', async () => { + // The state a deploy leaves behind: every process starts with an empty store, so the requests + // that arrive first all miss it at once. They join the load already running instead of each + // starting an aggregation of their own — the property this read depends on and does not + // implement itself, so it is asserted here rather than assumed. + let load: (log: Log) => void; + const getLatestFinancialLogSpy = jest + .spyOn(logService, 'getLatestFinancialLog') + .mockReturnValue(new Promise((resolve) => (load = resolve))); + + const first = service.getLatestBalance(); + const second = service.getLatestBalance(); + + load(logEntry()); + + await expect(first).resolves.toMatchObject({ timestamp: new Date('2026-07-14T12:00:00Z') }); + await expect(second).resolves.toMatchObject({ timestamp: new Date('2026-07-14T12:00:00Z') }); + expect(getLatestFinancialLogSpy).toHaveBeenCalledTimes(1); + }); + + it('serves the aggregate it holds when a later load fails', async () => { + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(logEntry()); + const loaded = await service.getLatestBalance(); + expect(loaded).toMatchObject({ timestamp: new Date('2026-07-14T12:00:00Z') }); + + // Nothing ages the entry out here, so force the load the store would do a minute later. + jest.spyOn(logService, 'getLatestFinancialLog').mockRejectedValue(new Error('database unavailable')); + + await expect(service.refreshLatestBalance()).rejects.toThrow('database unavailable'); + await expect(service.getLatestBalance()).resolves.toBe(loaded); }); }); - describe('setLatestBalance (aggregation write-through)', () => { - it('aggregates byType / byBlockchain (Scrypt split, 5000 CHF Other thresholds) and writes the result into the store', () => { + describe('refreshLatestBalance (aggregation into the store)', () => { + it('aggregates byType / byBlockchain (Scrypt split, 5000 CHF Other thresholds) and writes the result into the store', async () => { // Fixture designed so every aggregation branch is exercised once: // // byType: @@ -418,12 +489,12 @@ describe('DashboardFinancialService', () => { ], }; - service.setLatestBalance(timestamp, assetLog, balancesByFinancialType, assets); + await refreshFrom(timestamp, assetLog, balancesByFinancialType, assets); - expect(latestBalanceStore.set).toHaveBeenCalledWith(expected); + await expect(service.getLatestBalance()).resolves.toEqual(expected); }); - it('treats a priceless asset (priceChf: null) neutrally: neither its own blockchain group nor a shared one is skewed', () => { + it('treats a priceless asset (priceChf: null) neutrally: neither its own blockchain group nor a shared one is skewed', async () => { // asset.approxPriceChf is a nullable `double precision` column in production (144 of 430 rows // are NULL there; 136 of those have no financialType at all) -- the AssetLog[id].priceChf: number // type does not reflect that. This is a regression guard for the round-trip removed in this PR: @@ -478,9 +549,62 @@ describe('DashboardFinancialService', () => { ], }; - service.setLatestBalance(timestamp, assetLog, balancesByFinancialType, assets); + await refreshFrom(timestamp, assetLog, balancesByFinancialType, assets); + + await expect(service.getLatestBalance()).resolves.toEqual(expected); + }); + + it('keeps the aggregate it holds when the newest entry cannot be parsed', async () => { + // The job runs every minute. A single malformed entry must not replace a good value with + // nothing, and the endpoint must keep answering while someone looks at the entry. + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(logEntry()); + const loaded = await service.getLatestBalance(); + expect(loaded).toMatchObject({ timestamp: new Date('2026-07-14T12:00:00Z') }); + + jest + .spyOn(logService, 'getLatestFinancialLog') + .mockResolvedValue({ id: 2, created: new Date(), message: 'not json' } as Log); + + await expect(service.refreshLatestBalance()).rejects.toThrow('id 2'); + await expect(service.getLatestBalance()).resolves.toBe(loaded); + }); - expect(latestBalanceStore.set).toHaveBeenCalledWith(expected); + it('leaves the store empty when there is no log entry yet', async () => { + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(undefined); + + await service.refreshLatestBalance(); + + await expect(service.getLatestBalance()).resolves.toBeUndefined(); + }); + + it('does not query assets when the log entry holds none', async () => { + jest + .spyOn(logService, 'getLatestFinancialLog') + .mockResolvedValue({ id: 1, created: new Date(), message: JSON.stringify({}) } as Log); + const getAssetsByIdSpy = jest.spyOn(assetService, 'getAssetsById'); + + await service.refreshLatestBalance(); + + expect(getAssetsByIdSpy).not.toHaveBeenCalled(); + await expect(service.getLatestBalance()).resolves.toBeDefined(); + }); + + it('replaces an entry the read would still have served', async () => { + // What the job is for: the request path in this process finds a current value instead of + // loading one itself. + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue(logEntry()); + await service.getLatestBalance(); + + jest.spyOn(logService, 'getLatestFinancialLog').mockResolvedValue({ + id: 2, + created: new Date('2026-07-14T12:01:00Z'), + message: JSON.stringify({ assets: {}, balancesByFinancialType: {} }), + } as Log); + await service.refreshLatestBalance(); + + await expect(service.getLatestBalance()).resolves.toMatchObject({ + timestamp: new Date('2026-07-14T12:01:00Z'), + }); }); }); }); diff --git a/src/subdomains/supporting/dashboard/dashboard-financial.service.ts b/src/subdomains/supporting/dashboard/dashboard-financial.service.ts index 8a71b34f5c..b5344a13e5 100644 --- a/src/subdomains/supporting/dashboard/dashboard-financial.service.ts +++ b/src/subdomains/supporting/dashboard/dashboard-financial.service.ts @@ -1,8 +1,11 @@ import { Injectable } from '@nestjs/common'; +import { CronExpression } from '@nestjs/schedule'; import { Asset } from 'src/shared/models/asset/asset.entity'; import { AssetService } from 'src/shared/models/asset/asset.service'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { RefRewardService } from '../../core/referral/reward/services/ref-reward.service'; -import { AssetLog, BalancesByFinancialType } from '../log/dto/log.dto'; +import { AssetLog, BalancesByFinancialType, FinanceLog } from '../log/dto/log.dto'; import { Log } from '../log/log.entity'; import { FinancialLogSummary } from '../log/log.repository'; import { LogService } from '../log/log.service'; @@ -121,32 +124,58 @@ export class DashboardFinancialService { } } + /** + * Answers from LatestBalanceStore, which loads through `loadLatestBalance` when it has no entry + * or the one it holds has aged out. The load is what makes this correct in a process that never + * runs the refresh below - see the store for why there is such a process. + */ async getLatestBalance(): Promise { - return this.latestBalanceStore.get(); + return this.latestBalanceStore.get(() => this.loadLatestBalance()); } /** - * Called once a minute by LogJobService, immediately after it writes the FinancialDataLog entry - * these values are derived from. Builds the same response GET /v1/dashboard/financial/latest used - * to compute per-request (see buildLatestBalance below) and puts it in LatestBalanceStore, so the - * endpoint never touches the database again. Synchronous and DB-free by construction: assetLog, - * balancesByFinancialType and assets are exactly what the caller already holds in memory from the - * same run. A failure in here must never propagate into the caller's equity/safety-mode path — - * that isolation is the caller's responsibility (its own try/catch around this call), not this - * method's. + * Keeps the entry in LatestBalanceStore warm, so that GET /v1/dashboard/financial/latest finds a + * current value in this process instead of loading one itself. + * + * Scope Api, because the store it fills is a field of this service and its reader is the endpoint + * above. It only reads: LogJobService writes the entry, this parses it and aggregates - once a + * minute, outside any request. Refresh only, never the sole filler: the job is leased, so in a + * deployment with several API processes it runs in one of them per tick, and the request path is + * what fills the rest. + * + * Every minute rather than the 15 minutes CONTRIBUTING prefers, because that is the interval + * LogJobService already writes the underlying entry at (TRADING_LOG, EVERY_MINUTE). A longer + * one here would not save a write, it would only serve a staler value than the data allows. */ - setLatestBalance( - timestamp: Date, - assetLog: AssetLog, - balancesByFinancialType: BalancesByFinancialType, - assets: Asset[], - ): void { - this.latestBalanceStore.set(this.buildLatestBalance(timestamp, assetLog, balancesByFinancialType, assets)); + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.API, process: Process.LATEST_BALANCE_CACHE }) + async refreshLatestBalance(): Promise { + await this.latestBalanceStore.refresh(() => this.loadLatestBalance()); + } + + /** + * Builds the response from the most recent FinancialDataLog entry. `undefined` when there is no + * entry at all, which is the honest answer for a database that has never had one. + */ + private async loadLatestBalance(): Promise { + const latest = await this.logService.getLatestFinancialLog(); + if (!latest) return undefined; + + let financeLog: FinanceLog; + try { + financeLog = JSON.parse(latest.message); + } catch (e) { + // Raised rather than returned as `undefined`: the store keeps the aggregate it has, so one + // malformed entry does not replace a good value with nothing. The refresh job reports it. + throw new Error(`Failed to parse the latest financial log (id ${latest.id})`, { cause: e }); + } + + const assets = financeLog.assets + ? await this.assetService.getAssetsById(Object.keys(financeLog.assets).map(Number)) + : []; + + return this.buildLatestBalance(latest.created, financeLog.assets, financeLog.balancesByFinancialType, assets); } - // Unchanged aggregation that used to run inline in getLatestBalance against a freshly parsed - // FinancialDataLog message and a database asset lookup: identical logic, moved here verbatim: only - // its inputs changed (passed in directly instead of JSON.parse(latest.message) / assetService.getAssetsById). private buildLatestBalance( timestamp: Date, assetLog: AssetLog, diff --git a/src/subdomains/supporting/dashboard/latest-balance.store.ts b/src/subdomains/supporting/dashboard/latest-balance.store.ts index 738fb3be9a..d38c7e3935 100644 --- a/src/subdomains/supporting/dashboard/latest-balance.store.ts +++ b/src/subdomains/supporting/dashboard/latest-balance.store.ts @@ -1,24 +1,43 @@ import { Injectable } from '@nestjs/common'; +import { AsyncCache, CacheItemResetPeriod } from 'src/shared/utils/async-cache'; import { LatestBalanceResponseDto } from './dto/financial-log.dto'; +const LATEST_BALANCE_KEY = 'latest'; + /** - * Holds the single most recent LatestBalanceResponseDto, written once a minute by LogJobService - * right after it writes the FinancialDataLog entry the value is derived from, and read by - * GET /v1/dashboard/financial/latest. Exactly one entry, replaced wholesale on every job run: no - * TTL, no eviction, no size cap. There is only ever one API process instance and the writing cron - * job holds a lock, so there is never more than one writer and no cross-instance state to reconcile. - * Empty (undefined) until the first job run after process start; the read side must not fall back - * to the database in that window (see DashboardFinancialService.getLatestBalance). + * Holds the single most recent LatestBalanceResponseDto, derived from the newest FinancialDataLog + * entry and read by GET /v1/dashboard/financial/latest. Exactly one entry, replaced wholesale: no + * eviction, no size cap. + * + * The store is process-local, so every process answering that request needs its own copy - and the + * job cannot be what puts it there. DashboardFinancialService.refreshLatestBalance is scoped `api` + * and therefore runs under a lease: with more than one API process, one of them takes the tick and + * the others do not run the job at all. A store that only the job filled would stay at whatever + * the losing processes started with. + * + * So the read fills it: `get` loads through the loader it is given whenever there is no entry or + * the one it holds has aged out - what CONTRIBUTING asks of a cache a request path reads. The job keeps + * the entry warm in the process that took the tick, so requests there never wait for the load. */ @Injectable() export class LatestBalanceStore { - private value: LatestBalanceResponseDto | undefined; + private readonly cache = new AsyncCache(CacheItemResetPeriod.EVERY_1_MINUTE); - get(): LatestBalanceResponseDto | undefined { - return this.value; + /** + * Serves the entry, loading it through `load` when there is none or it has aged out. A failing + * load leaves whatever is there in place and is not raised at the request: an aggregate a minute + * older answers better than an error, and the refresh below is what reports the failure. + * + * Requests that miss together share the one load: `AsyncCache` keeps the running update on the + * entry and hands it to everyone who asks while it is in flight. That is what a restart depends + * on — the store starts empty in every process, so the requests arriving first all miss at once. + */ + async get(load: () => Promise): Promise { + return this.cache.get(LATEST_BALANCE_KEY, load, undefined, true); } - set(value: LatestBalanceResponseDto): void { - this.value = value; + /** Replaces the entry regardless of its age, and raises what `load` throws so the job reports it. */ + async refresh(load: () => Promise): Promise { + await this.cache.get(LATEST_BALANCE_KEY, load, () => true); } } diff --git a/src/subdomains/supporting/dex/services/dex.service.ts b/src/subdomains/supporting/dex/services/dex.service.ts index 168314cdfb..15902703f8 100644 --- a/src/subdomains/supporting/dex/services/dex.service.ts +++ b/src/subdomains/supporting/dex/services/dex.service.ts @@ -4,7 +4,8 @@ import { FeeAmount } from '@uniswap/v3-sdk'; import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; import { Asset } from 'src/shared/models/asset/asset.entity'; import { DfxLogger, LogLevel } from 'src/shared/services/dfx-logger'; -import { DfxCron } from 'src/shared/utils/cron'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; @@ -322,7 +323,11 @@ export class DexService { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_30_SECONDS, { timeout: 1800 }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { + scope: CronScope.WORKER, + process: Process.DEX_PURCHASE_ORDER, + timeout: 1800, + }) async finalizePurchaseOrders(): Promise { await this.alertStrandedPurchaseOrders(); diff --git a/src/subdomains/supporting/fiat-output/fiat-output-frick.service.ts b/src/subdomains/supporting/fiat-output/fiat-output-frick.service.ts index cf267260cd..3c986daf7f 100644 --- a/src/subdomains/supporting/fiat-output/fiat-output-frick.service.ts +++ b/src/subdomains/supporting/fiat-output/fiat-output-frick.service.ts @@ -12,7 +12,7 @@ import { BankFrickService } from 'src/integration/bank/services/frick.service'; import { IbanService } from 'src/integration/bank/services/iban.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { FindOptionsWhere, In, IsNull, Not } from 'typeorm'; import { IbanBankName } from '../bank/bank/dto/bank.dto'; import { FiatOutput, TransactionCharge } from './fiat-output.entity'; @@ -28,7 +28,7 @@ export class FiatOutputFrickService { private readonly ibanService: IbanService, ) {} - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.FIAT_OUTPUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.FIAT_OUTPUT, timeout: 1800 }) async checkFrickOrderStatus(): Promise { if (DisabledProcess(Process.FIAT_OUTPUT_FRICK_STATUS_CHECK)) return; if (!this.frickService.isAvailable()) return; diff --git a/src/subdomains/supporting/fiat-output/fiat-output-job.service.ts b/src/subdomains/supporting/fiat-output/fiat-output-job.service.ts index dbdc2db21c..b457e33daf 100644 --- a/src/subdomains/supporting/fiat-output/fiat-output-job.service.ts +++ b/src/subdomains/supporting/fiat-output/fiat-output-job.service.ts @@ -15,7 +15,7 @@ import { Country } from 'src/shared/models/country/country.entity'; import { CountryService } from 'src/shared/models/country/country.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { FindOptionsWhere, In, IsNull, Like, Not } from 'typeorm'; import { BankTxRepeatService } from '../bank-tx/bank-tx-repeat/bank-tx-repeat.service'; @@ -64,7 +64,7 @@ export class FiatOutputJobService { private readonly bankService: BankService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.FIAT_OUTPUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.FIAT_OUTPUT, timeout: 1800 }) async fillFiatOutput() { await this.assignBankAccount(); await this.setReadyDate(); @@ -77,7 +77,7 @@ export class FiatOutputJobService { await this.notifyScryptDeposits(); } - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.FIAT_OUTPUT }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.FIAT_OUTPUT }) async checkOlkypayOrderStatus(): Promise { if (DisabledProcess(Process.FIAT_OUTPUT_OLKYPAY_STATUS_CHECK)) return; if (!this.olkypayService.isAvailable()) return; @@ -103,7 +103,7 @@ export class FiatOutputJobService { } } - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.FIAT_OUTPUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.FIAT_OUTPUT, timeout: 1800 }) async generateReports() { const entities = await this.fiatOutputRepo.find({ where: { reportCreated: false, isComplete: true }, diff --git a/src/subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts b/src/subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts index c56e71ed63..188c4c5793 100644 --- a/src/subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts +++ b/src/subdomains/supporting/fiat-payin/services/fiat-payin-sync.service.ts @@ -6,7 +6,7 @@ import { ChargebackReason, ChargebackState, TransactionStatus } from 'src/integr import { SiftService } from 'src/integration/sift/services/sift.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BuyService } from 'src/subdomains/core/buy-crypto/routes/buy/buy.service'; import { TransactionSourceType } from '../../payment/entities/transaction.entity'; @@ -32,7 +32,7 @@ export class FiatPayInSyncService { // --- JOBS --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.FIAT_PAY_IN, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.FIAT_PAY_IN, timeout: 1800 }) async syncCheckout() { if (!this.checkoutService.isAvailable()) { if (!this.unavailableWarningLogged) { diff --git a/src/subdomains/supporting/log/__tests__/log-job.service.spec.ts b/src/subdomains/supporting/log/__tests__/log-job.service.spec.ts index daf001fa7a..ec8304ea4a 100644 --- a/src/subdomains/supporting/log/__tests__/log-job.service.spec.ts +++ b/src/subdomains/supporting/log/__tests__/log-job.service.spec.ts @@ -35,7 +35,6 @@ import { Bank } from '../../bank/bank/bank.entity'; import { BankService } from '../../bank/bank/bank.service'; import { frickCHF, frickEUR, olkyEUR, yapealCHF, yapealEUR } from '../../bank/bank/__mocks__/bank.entity.mock'; import { IbanBankName } from '../../bank/bank/dto/bank.dto'; -import { DashboardFinancialService } from '../../dashboard/dashboard-financial.service'; import { createCustomFiatOutput } from '../../fiat-output/__mocks__/fiat-output.entity.mock'; import { createCustomCryptoInput } from '../../payin/entities/__mocks__/crypto-input.entity.mock'; import { PayInService } from '../../payin/services/payin.service'; @@ -66,7 +65,6 @@ describe('LogJobService', () => { let payoutService: PayoutService; let processService: ProcessService; let paymentBalanceService: PaymentBalanceService; - let dashboardFinancialService: DashboardFinancialService; beforeEach(async () => { tradingRuleService = createMock(); @@ -89,7 +87,6 @@ describe('LogJobService', () => { payoutService = createMock(); processService = createMock(); paymentBalanceService = createMock(); - dashboardFinancialService = createMock(); const module: TestingModule = await Test.createTestingModule({ imports: [TestSharedModule], @@ -115,7 +112,6 @@ describe('LogJobService', () => { { provide: PayoutService, useValue: payoutService }, { provide: ProcessService, useValue: processService }, { provide: PaymentBalanceService, useValue: paymentBalanceService }, - { provide: DashboardFinancialService, useValue: dashboardFinancialService }, TestUtil.provideConfig(), ], }).compile(); @@ -496,46 +492,6 @@ describe('LogJobService', () => { }); }); - describe('latest-balance cache write isolation (cache failure must not arm the equity safety mode)', () => { - // a healthy, finite book comfortably above the minimum -> the equity path leaves safety mode off - function setup() { - jest.spyOn(service as any, 'getTradingLog').mockResolvedValue({}); - jest.spyOn(service as any, 'getAssetLog').mockResolvedValue({}); - jest - .spyOn(service as any, 'getBalancesByFinancialType') - .mockReturnValue({ EUR: { plusBalance: 5000, plusBalanceChf: 5000, minusBalance: 0, minusBalanceChf: 0 } }); - jest.spyOn(service as any, 'getChangeLog').mockResolvedValue({}); - jest.spyOn(assetService, 'getAssetsWith').mockResolvedValue([] as any); - jest.spyOn(settingService, 'getObj').mockResolvedValue(100 as any); - jest.spyOn(refRewardService, 'getOpenRefCreditLiability').mockResolvedValue({ amountEur: 0, amountChf: 0 }); - jest - .spyOn(logService, 'maxEntity') - .mockResolvedValue({ message: JSON.stringify({ balancesTotal: { totalBalanceChf: 5000 } }) } as any); - // created is read as financialDataLog.created for the write-through cache call - jest.spyOn(logService, 'create').mockResolvedValue({ created: new Date('2026-07-14T12:00:00Z') } as any); - } - - it('keeps safety mode off, logs loudly and still resolves when setLatestBalance throws', async () => { - const errorSpy = jest.spyOn(service['logger'], 'error'); - setup(); - jest.spyOn(dashboardFinancialService, 'setLatestBalance').mockImplementation(() => { - throw new Error('cache write failed'); - }); - - await expect(service.saveTradingLog()).resolves.toBeUndefined(); - - // equity path already set safety mode correctly for the healthy book; the cache catch must not - // rethrow into the outer catch that would arm safety mode - expect(processService.setSafetyModeActive).toHaveBeenCalledWith(false); - expect(processService.setSafetyModeActive).not.toHaveBeenCalledWith(true); - - expect(errorSpy).toHaveBeenCalledWith( - 'Failed to update the latest-balance cache for the dashboard', - expect.any(Error), - ); - }); - }); - describe('safety mode (fail closed on non-finite total)', () => { function setup(buckets: Record, minTotalBalanceChf: number) { jest.spyOn(service as any, 'getTradingLog').mockResolvedValue({}); diff --git a/src/subdomains/supporting/log/log-job.module.ts b/src/subdomains/supporting/log/log-job.module.ts index 9fce08afb3..10c25ab0f9 100644 --- a/src/subdomains/supporting/log/log-job.module.ts +++ b/src/subdomains/supporting/log/log-job.module.ts @@ -10,7 +10,6 @@ import { SellCryptoModule } from 'src/subdomains/core/sell-crypto/sell-crypto.mo import { TradingModule } from 'src/subdomains/core/trading/trading.module'; import { BankTxModule } from '../bank-tx/bank-tx.module'; import { BankModule } from '../bank/bank.module'; -import { DashboardModule } from '../dashboard/dashboard.module'; import { PayInModule } from '../payin/payin.module'; import { PayoutModule } from '../payout/payout.module'; import { LogJobService } from './log-job.service'; @@ -32,7 +31,6 @@ import { LogModule } from './log.module'; ReferralModule, PayoutModule, PaymentLinkPaymentModule, - DashboardModule, ], controllers: [], providers: [LogJobService], diff --git a/src/subdomains/supporting/log/log-job.service.ts b/src/subdomains/supporting/log/log-job.service.ts index f85f7037b0..63b1ee8e91 100644 --- a/src/subdomains/supporting/log/log-job.service.ts +++ b/src/subdomains/supporting/log/log-job.service.ts @@ -15,7 +15,7 @@ import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process, ProcessService } from 'src/shared/services/process.service'; import { AsyncCache, CacheItemResetPeriod } from 'src/shared/utils/async-cache'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { AmountType, Util } from 'src/shared/utils/util'; import { BuyCrypto } from 'src/subdomains/core/buy-crypto/process/entities/buy-crypto.entity'; import { BuyCryptoService } from 'src/subdomains/core/buy-crypto/process/services/buy-crypto.service'; @@ -40,7 +40,6 @@ import { BankTx, BankTxIndicator, BankTxType } from '../bank-tx/bank-tx/entities import { BankTxService } from '../bank-tx/bank-tx/services/bank-tx.service'; import { BankService } from '../bank/bank/bank.service'; import { IbanBankName } from '../bank/bank/dto/bank.dto'; -import { DashboardFinancialService } from '../dashboard/dashboard-financial.service'; import { CryptoInput } from '../payin/entities/crypto-input.entity'; import { PayInService } from '../payin/services/payin.service'; import { PayoutOrder, PayoutOrderContext } from '../payout/entities/payout-order.entity'; @@ -111,10 +110,9 @@ export class LogJobService { private readonly payoutService: PayoutService, private readonly processService: ProcessService, private readonly paymentBalanceService: PaymentBalanceService, - private readonly dashboardFinancialService: DashboardFinancialService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.TRADING_LOG, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.TRADING_LOG, timeout: 1800 }) async saveTradingLog() { try { // trading log @@ -191,7 +189,7 @@ export class LogJobService { const btcAssetPriceChf = btcAsset ? assetLog[btcAsset.id]?.priceChf : undefined; const btcPriceChfColumn = btcAssetPriceChf != null && Number.isFinite(btcAssetPriceChf) ? btcAssetPriceChf : null; - const financialDataLog = await this.logService.create({ + await this.logService.create({ system: 'LogService', subsystem: 'FinancialDataLog', severity: LogSeverity.INFO, @@ -225,21 +223,6 @@ export class LogJobService { category: null, }); - // Write-through for GET /v1/dashboard/financial/latest: precompute here so that endpoint never - // touches the database or re-parses this message. Independent of the equity path above (which - // has already run and already armed/disarmed the safety mode correctly), so a failure here must - // never escalate to that switch: own try/catch, log loudly, never rethrow. - try { - this.dashboardFinancialService.setLatestBalance( - financialDataLog.created, - assetLog, - balancesByFinancialType, - assets, - ); - } catch (e) { - this.logger.error('Failed to update the latest-balance cache for the dashboard', e); - } - // The changeLog feeds only the informative FinancialChangesLog and is independent of the equity // path above, so it runs in its own try/catch: a reporting-price failure must not arm the equity // safety mode; the equity path above has already run and set it correctly. On failure we log the diff --git a/src/subdomains/supporting/log/log.service.ts b/src/subdomains/supporting/log/log.service.ts index de207e4126..1b9545d08d 100644 --- a/src/subdomains/supporting/log/log.service.ts +++ b/src/subdomains/supporting/log/log.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { CreateLogDto, LogCleanupSetting, UpdateLogDto } from './dto/create-log.dto'; import { SetFinancialLogValidityDto } from './dto/set-financial-log-validity.dto'; import { @@ -24,7 +24,7 @@ export class LogService { private readonly settingService: SettingService, ) {} - @DfxCron(CronExpression.EVERY_DAY_AT_11PM, { process: Process.LOG_CLEANUP }) + @DfxCron(CronExpression.EVERY_DAY_AT_11PM, { scope: CronScope.WORKER, process: Process.LOG_CLEANUP }) async cleanup(): Promise { const logCleanupSettings = await this.settingService.getObj('logCleanup', []); diff --git a/src/subdomains/supporting/notification/services/notification-job.service.ts b/src/subdomains/supporting/notification/services/notification-job.service.ts index 42b3db3442..68debf3fe0 100644 --- a/src/subdomains/supporting/notification/services/notification-job.service.ts +++ b/src/subdomains/supporting/notification/services/notification-job.service.ts @@ -3,7 +3,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { LessThanOrEqual } from 'typeorm'; import { MailFactory } from '../factories/mail.factory'; @@ -33,7 +33,7 @@ export class NotificationJobService { private readonly mailService: MailService, ) {} - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.MAIL_RETRY, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.MAIL_RETRY, timeout: 7200 }) async resendUncompletedMails(): Promise { const uncompletedMails = await this.notificationRepo.find({ where: { isComplete: false, created: LessThanOrEqual(Util.minutesBefore(1)) }, diff --git a/src/subdomains/supporting/payin/services/payin-notification.service.ts b/src/subdomains/supporting/payin/services/payin-notification.service.ts index b2654afc10..a7ebb21cd4 100644 --- a/src/subdomains/supporting/payin/services/payin-notification.service.ts +++ b/src/subdomains/supporting/payin/services/payin-notification.service.ts @@ -2,7 +2,7 @@ import { Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailFactory, @@ -23,7 +23,7 @@ export class PayInNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.PAY_IN_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.PAY_IN_MAIL, timeout: 1800 }) async sendNotificationMails(): Promise { await this.returnedCryptoInput(); } diff --git a/src/subdomains/supporting/payin/services/payin.service.ts b/src/subdomains/supporting/payin/services/payin.service.ts index 0e3be09c49..b9c7440323 100644 --- a/src/subdomains/supporting/payin/services/payin.service.ts +++ b/src/subdomains/supporting/payin/services/payin.service.ts @@ -6,7 +6,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; import { Swap } from 'src/subdomains/core/buy-crypto/routes/swap/swap.entity'; @@ -326,26 +326,26 @@ export class PayInService { // --- JOBS --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async forwardPayInEntries(): Promise { await this.forwardPayIns(); await this.processStrandedSendingPayIns(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async returnPayInEntries(): Promise { await this.returnPayIns(); await this.processStrandedSendingPayIns(); } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkConfirmations(): Promise { await this.checkInputConfirmations(); await this.checkOutputConfirmations(); await this.checkReturnConfirmations(); } - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async updateFailedPayments(): Promise { const checkDate = Util.minutesBefore(15); diff --git a/src/subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts index fb253de7d7..3b2bc06aee 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/base/citrea.strategy.ts @@ -6,7 +6,7 @@ import { Asset, AssetType } from 'src/shared/models/asset/asset.entity'; import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { TransactionRequestService } from 'src/subdomains/supporting/payment/services/transaction-request.service'; import { PayInType } from '../../../../entities/crypto-input.entity'; @@ -37,7 +37,7 @@ export abstract class CitreaBaseStrategy extends RegisterStrategy { } // --- JOBS --- // - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { const activeDepositAddresses = await this.transactionRequestService.getActiveDepositAddresses( Util.hoursBefore(1), diff --git a/src/subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts index e8c34daffa..d04dcdad53 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/bitcoin.strategy.ts @@ -6,7 +6,7 @@ import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.e import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { DepositService } from 'src/subdomains/supporting/address-pool/deposit/deposit.service'; import { PayInType } from '../../../entities/crypto-input.entity'; @@ -31,7 +31,7 @@ export class BitcoinStrategy extends PollingStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_SECOND, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_SECOND, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { if (!this.payInBitcoinService.isAvailable()) { if (!this.unavailableWarningLogged) { diff --git a/src/subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts index a5f00c7f03..65f9dc4e84 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/cardano.strategy.ts @@ -8,7 +8,7 @@ import { Asset, AssetType } from 'src/shared/models/asset/asset.entity'; import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { TransactionRequestService } from 'src/subdomains/supporting/payment/services/transaction-request.service'; import { PayInType } from '../../../entities/crypto-input.entity'; @@ -41,7 +41,7 @@ export class CardanoStrategy extends RegisterStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { // not configured (no Tatum API key) -> skip, warn once if (!this.payInCardanoService.isConfigured) { diff --git a/src/subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts index cfe6551888..8738dee313 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/firo.strategy.ts @@ -6,7 +6,7 @@ import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.e import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { DepositService } from 'src/subdomains/supporting/address-pool/deposit/deposit.service'; import { PayInType } from '../../../entities/crypto-input.entity'; @@ -31,7 +31,7 @@ export class FiroStrategy extends PollingStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_SECOND, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_SECOND, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { if (!this.payInFiroService.isAvailable()) { if (!this.unavailableWarningLogged) { diff --git a/src/subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts index 7843ab5541..4fcb631fcf 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/icp.strategy.ts @@ -8,7 +8,7 @@ import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { DepositService } from 'src/subdomains/supporting/address-pool/deposit/deposit.service'; import { PayInType } from '../../../entities/crypto-input.entity'; import { PayInEntry } from '../../../interfaces'; @@ -40,7 +40,7 @@ export class InternetComputerStrategy extends RegisterStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { const allDeposits = await this.depositService.getUsedDepositsByBlockchain(this.blockchain); const allDepositAddresses = allDeposits.map((d) => d.address); diff --git a/src/subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts index dfbea80073..3b957e53ca 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/monero.strategy.ts @@ -6,7 +6,7 @@ import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.e import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { PayInType } from '../../../entities/crypto-input.entity'; import { PayInEntry } from '../../../interfaces'; @@ -26,7 +26,7 @@ export class MoneroStrategy extends PollingStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_SECOND, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_SECOND, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { return super.checkPayInEntries(); } diff --git a/src/subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts b/src/subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts index 8db0d38816..976018ff00 100644 --- a/src/subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts +++ b/src/subdomains/supporting/payin/strategies/register/impl/zano.strategy.ts @@ -8,7 +8,7 @@ import { Asset } from 'src/shared/models/asset/asset.entity'; import { BlockchainAddress } from 'src/shared/models/blockchain-address'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { PayInType } from '../../../entities/crypto-input.entity'; import { PayInEntry } from '../../../interfaces'; @@ -28,7 +28,7 @@ export class ZanoStrategy extends PollingStrategy { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_SECOND, { process: Process.PAY_IN, timeout: 7200 }) + @DfxCron(CronExpression.EVERY_SECOND, { scope: CronScope.WORKER, process: Process.PAY_IN, timeout: 7200 }) async checkPayInEntries(): Promise { return super.checkPayInEntries(); } diff --git a/src/subdomains/supporting/payment/services/fee.service.ts b/src/subdomains/supporting/payment/services/fee.service.ts index 401b422333..4da3ca1fea 100644 --- a/src/subdomains/supporting/payment/services/fee.service.ts +++ b/src/subdomains/supporting/payment/services/fee.service.ts @@ -16,7 +16,7 @@ import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger, LogLevel } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { AccountType } from 'src/subdomains/generic/user/models/user-data/account-type.enum'; import { UserData } from 'src/subdomains/generic/user/models/user-data/user-data.entity'; @@ -88,7 +88,11 @@ export class FeeService { ) {} // --- JOBS --- // - @DfxCron(CronExpression.EVERY_10_MINUTES, { process: Process.BLOCKCHAIN_FEE_UPDATE, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_10_MINUTES, { + scope: CronScope.WORKER, + process: Process.BLOCKCHAIN_FEE_UPDATE, + timeout: 1800, + }) async updateBlockchainFees() { const blockchainFees = await this.blockchainFeeRepo.find({ relations: { asset: true } }); diff --git a/src/subdomains/supporting/payment/services/transaction-helper.ts b/src/subdomains/supporting/payment/services/transaction-helper.ts index b848730a9d..f1370577d8 100644 --- a/src/subdomains/supporting/payment/services/transaction-helper.ts +++ b/src/subdomains/supporting/payment/services/transaction-helper.ts @@ -17,7 +17,7 @@ import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; import { AsyncCache, CacheItemResetPeriod } from 'src/shared/utils/async-cache'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { AmountType, Util } from 'src/shared/utils/util'; import { AmlRule } from 'src/subdomains/core/aml/enums/aml-rule.enum'; import { AmlHelperService } from 'src/subdomains/core/aml/services/aml-helper.service'; @@ -87,7 +87,7 @@ export class TransactionHelper implements OnModuleInit { void this.updateCache(); } - @DfxCron(CronExpression.EVERY_5_MINUTES) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.BOTH }) async updateCache() { this.transactionSpecifications = await this.specRepo.find(); } diff --git a/src/subdomains/supporting/payment/services/transaction-notification.service.ts b/src/subdomains/supporting/payment/services/transaction-notification.service.ts index 9044b1312a..0b63c82a04 100644 --- a/src/subdomains/supporting/payment/services/transaction-notification.service.ts +++ b/src/subdomains/supporting/payment/services/transaction-notification.service.ts @@ -2,7 +2,7 @@ import { forwardRef, Inject, Injectable } from '@nestjs/common'; import { CronExpression } from '@nestjs/schedule'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BuyCrypto } from 'src/subdomains/core/buy-crypto/process/entities/buy-crypto.entity'; import { BuyFiat } from 'src/subdomains/core/sell-crypto/process/buy-fiat.entity'; @@ -27,7 +27,7 @@ export class TransactionNotificationService { private readonly bankTxService: BankTxService, ) {} - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.TX_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.TX_MAIL, timeout: 1800 }) async sendNotificationMails(): Promise { await this.txAssigned(); if (!DisabledProcess(Process.TX_UNASSIGNED_MAIL)) await this.txUnassigned(); diff --git a/src/subdomains/supporting/payment/services/transaction-request.service.ts b/src/subdomains/supporting/payment/services/transaction-request.service.ts index a5505a760c..162e962f2e 100644 --- a/src/subdomains/supporting/payment/services/transaction-request.service.ts +++ b/src/subdomains/supporting/payment/services/transaction-request.service.ts @@ -1,13 +1,13 @@ import { ForbiddenException, Inject, Injectable, NotFoundException, forwardRef } from '@nestjs/common'; -import { Cron, CronExpression } from '@nestjs/schedule'; +import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { Blockchain } from 'src/integration/blockchain/shared/enums/blockchain.enum'; import { SiftService } from 'src/integration/sift/services/sift.service'; import { AssetService } from 'src/shared/models/asset/asset.service'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; -import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { Lock } from 'src/shared/utils/lock'; +import { Process } from 'src/shared/services/process.service'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { BuyService } from 'src/subdomains/core/buy-crypto/routes/buy/buy.service'; import { BuyPaymentInfoDto } from 'src/subdomains/core/buy-crypto/routes/buy/dto/buy-payment-info.dto'; @@ -49,20 +49,26 @@ export class TransactionRequestService { private readonly swapService: SwapService, ) {} - @Cron(CronExpression.EVERY_MINUTE) - @Lock(7200) + // useDelay: false keeps the schedule this job had as a native @Cron. DfxCron staggers job + // starts by default, which for a minute-based expression spreads them over up to 30 seconds - + // moving it here would change when it runs, not just how it is registered. + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.TX_REQUEST, + useDelay: false, + timeout: 7200, + }) async txRequestStatusSync() { - if (DisabledProcess(Process.TX_REQUEST)) return; - await this.syncStatus(); await this.deleteOldTxRequests(); } - @Cron(CronExpression.EVERY_DAY_AT_3AM) - @Lock(7200) + @DfxCron(CronExpression.EVERY_DAY_AT_3AM, { + scope: CronScope.WORKER, + process: Process.TX_REQUEST_WAITING_EXPIRY, + timeout: 7200, + }) async txRequestWaitingExpiryCheck() { - if (DisabledProcess(Process.TX_REQUEST_WAITING_EXPIRY)) return; - const expiryDate = Util.daysBefore(Config.txRequestWaitingExpiryDays); const entities = await this.transactionRequestRepo.findBy({ status: TransactionRequestStatus.WAITING_FOR_PAYMENT, diff --git a/src/subdomains/supporting/payout/services/payout.service.ts b/src/subdomains/supporting/payout/services/payout.service.ts index c078f8ef3a..87588ccc46 100644 --- a/src/subdomains/supporting/payout/services/payout.service.ts +++ b/src/subdomains/supporting/payout/services/payout.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Asset } from 'src/shared/models/asset/asset.entity'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { DisabledProcess, Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { NotificationService } from 'src/subdomains/supporting/notification/services/notification.service'; @@ -173,7 +173,7 @@ export class PayoutService { } //*** JOBS ***// - @DfxCron(CronExpression.EVERY_30_SECONDS, { process: Process.PAY_OUT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_30_SECONDS, { scope: CronScope.WORKER, process: Process.PAY_OUT, timeout: 1800 }) async processOrders(): Promise { await this.checkExistingOrders(); await this.prepareNewOrders(); diff --git a/src/subdomains/supporting/pricing/services/asset-prices-job.service.ts b/src/subdomains/supporting/pricing/services/asset-prices-job.service.ts index c6ca2ed6b6..9ac4e3e875 100644 --- a/src/subdomains/supporting/pricing/services/asset-prices-job.service.ts +++ b/src/subdomains/supporting/pricing/services/asset-prices-job.service.ts @@ -6,7 +6,7 @@ import { UpdateResult } from 'src/shared/models/entity'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger, LogLevel } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MoreThanOrEqual } from 'typeorm'; import { PriceInvalidException } from '../domain/exceptions/price-invalid.exception'; @@ -24,7 +24,7 @@ export class AssetPricesJobService { private readonly assetPriceRepo: AssetPriceRepository, ) {} - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.PRICING, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.PRICING, timeout: 3600 }) async updatePrices() { const assetsToUpdate = await this.assetService.getPricedAssets(); const updates: UpdateResult[] = []; @@ -59,7 +59,7 @@ export class AssetPricesJobService { await this.assetService.updateAssets(updates); } - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.PRICING, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.PRICING, timeout: 3600 }) async updatePaymentPrices() { const relevantFiats = await this.fiatService.getActiveFiat(); const relevantAssets = await this.assetService.getPaymentAssets(); diff --git a/src/subdomains/supporting/pricing/services/fiat-prices.service.ts b/src/subdomains/supporting/pricing/services/fiat-prices.service.ts index 102c52c712..d030f8a5fa 100644 --- a/src/subdomains/supporting/pricing/services/fiat-prices.service.ts +++ b/src/subdomains/supporting/pricing/services/fiat-prices.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { FiatService } from 'src/shared/models/fiat/fiat.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { PriceCurrency, PriceValidity, PricingService } from './pricing.service'; @Injectable() @@ -16,7 +16,7 @@ export class FiatPricesService { ) {} // --- JOBS --- // - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.PRICING, timeout: 3600 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.PRICING, timeout: 3600 }) async updatePrices() { const fiats = await this.fiatService.getActiveFiat(); diff --git a/src/subdomains/supporting/realunit/realunit-job.service.ts b/src/subdomains/supporting/realunit/realunit-job.service.ts index a16ea34ee3..515e5b411a 100644 --- a/src/subdomains/supporting/realunit/realunit-job.service.ts +++ b/src/subdomains/supporting/realunit/realunit-job.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { TransactionRequestService } from 'src/subdomains/supporting/payment/services/transaction-request.service'; import { HistoryEventDto } from './dto/realunit.dto'; @@ -21,7 +21,11 @@ export class RealUnitJobService { // Completes open REALU buy quotes as soon as the shares arrive on-chain. Share allocations // triggered outside the DFX payment flow (e.g. booked manually by the issuer) would otherwise // leave the quote in WaitingForPayment and keep showing a pending payment to the customer. - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.REALUNIT_QUOTE_COMPLETION, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { + scope: CronScope.WORKER, + process: Process.REALUNIT_QUOTE_COMPLETION, + timeout: 1800, + }) async completeSettledQuotes(): Promise { const realuAsset = await this.realunitService.getRealuAsset(); const openQuotes = await this.transactionRequestService.getOpenBuyQuotes(realuAsset.id); @@ -87,7 +91,11 @@ export class RealUnitJobService { // Resolves RealUnit W2W transfer requests stuck in PROCESSING after a crash/restart between the // atomic claim and the broadcast/callback in confirmTransfer — see // RealUnitService.reconcilePendingTransfers for the actual reconciliation logic. - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.REALUNIT_TRANSFER_RECONCILIATION, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { + scope: CronScope.WORKER, + process: Process.REALUNIT_TRANSFER_RECONCILIATION, + timeout: 1800, + }) async reconcilePendingTransfers(): Promise { await this.realunitService.reconcilePendingTransfers(); } diff --git a/src/subdomains/supporting/support-issue/services/limit-request-notification.service.ts b/src/subdomains/supporting/support-issue/services/limit-request-notification.service.ts index 6653c61dab..a97b66a5d8 100644 --- a/src/subdomains/supporting/support-issue/services/limit-request-notification.service.ts +++ b/src/subdomains/supporting/support-issue/services/limit-request-notification.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { MailContext, MailType } from 'src/subdomains/supporting/notification/enums'; import { MailKey, MailTranslationKey } from 'src/subdomains/supporting/notification/factories/mail.factory'; @@ -22,7 +22,11 @@ export class LimitRequestNotificationService { private readonly notificationService: NotificationService, ) {} - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.LIMIT_REQUEST_MAIL, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { + scope: CronScope.WORKER, + process: Process.LIMIT_REQUEST_MAIL, + timeout: 1800, + }) async sendNotificationMails(): Promise { await this.limitRequestAcceptedManual(); } diff --git a/src/subdomains/supporting/support-issue/services/support-escalation.service.ts b/src/subdomains/supporting/support-issue/services/support-escalation.service.ts index cfc8855cb5..0f9fac833a 100644 --- a/src/subdomains/supporting/support-issue/services/support-escalation.service.ts +++ b/src/subdomains/supporting/support-issue/services/support-escalation.service.ts @@ -5,7 +5,7 @@ import { SettingService } from 'src/shared/models/setting/setting.service'; import { DfxLogger } from 'src/shared/services/dfx-logger'; import { HttpService } from 'src/shared/services/http.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { In } from 'typeorm'; import { SupportIssueReasonLabelMap, SupportIssueTypeLabelMap } from '../dto/support-issue-label'; @@ -200,7 +200,7 @@ export class SupportEscalationService { // --- Escalation detection --- - @DfxCron(CronExpression.EVERY_5_MINUTES, { process: Process.SUPPORT_BOT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_5_MINUTES, { scope: CronScope.WORKER, process: Process.SUPPORT_BOT, timeout: 1800 }) async checkEscalations(): Promise { if (!this.token) return; diff --git a/src/subdomains/supporting/support-issue/services/support-issue-job.service.ts b/src/subdomains/supporting/support-issue/services/support-issue-job.service.ts index a9fd636bb3..53c500c419 100644 --- a/src/subdomains/supporting/support-issue/services/support-issue-job.service.ts +++ b/src/subdomains/supporting/support-issue/services/support-issue-job.service.ts @@ -3,7 +3,7 @@ import { CronExpression } from '@nestjs/schedule'; import { Config } from 'src/config/config'; import { SettingService } from 'src/shared/models/setting/setting.service'; import { Process } from 'src/shared/services/process.service'; -import { DfxCron } from 'src/shared/utils/cron'; +import { CronScope, DfxCron } from 'src/shared/utils/cron'; import { Util } from 'src/shared/utils/util'; import { CheckStatus } from 'src/subdomains/core/aml/enums/check-status.enum'; import { BuyCryptoStatus } from 'src/subdomains/core/buy-crypto/process/entities/buy-crypto.entity'; @@ -32,7 +32,7 @@ export class SupportIssueJobService { private readonly settingsService: SettingService, ) {} - @DfxCron(CronExpression.EVERY_HOUR, { process: Process.SUPPORT_BOT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_HOUR, { scope: CronScope.WORKER, process: Process.SUPPORT_BOT, timeout: 1800 }) async autoOnHold() { const entities = await this.supportIssueRepo.find({ where: { @@ -52,7 +52,7 @@ export class SupportIssueJobService { } } - @DfxCron(CronExpression.EVERY_MINUTE, { process: Process.SUPPORT_BOT, timeout: 1800 }) + @DfxCron(CronExpression.EVERY_MINUTE, { scope: CronScope.WORKER, process: Process.SUPPORT_BOT, timeout: 1800 }) async sendAutoResponses() { const disabledTemplates = await this.settingsService .get('supportBot') diff --git a/src/tracing.ts b/src/tracing.ts index 937f4f5fb1..1b92db8432 100644 --- a/src/tracing.ts +++ b/src/tracing.ts @@ -1,6 +1,13 @@ import { SpanKind, SpanStatusCode } from '@opentelemetry/api'; import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'; +import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'; +// Pinned to an exact version in package.json, not a range: the exporter and the NodeSDK below +// both depend on one exact @opentelemetry/sdk-metrics, so a range resolving higher installs a +// second copy beside theirs. The reader constructed here would then come from a different copy of +// the package than the one the SDK reads it as, and nothing in the build says so - the types are +// structural and match either way. Raise the pin together with @opentelemetry/sdk-node. +import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { NodeSDK } from '@opentelemetry/sdk-node'; import { BatchSpanProcessor, ReadableSpan, SpanProcessor } from '@opentelemetry/sdk-trace-base'; @@ -61,19 +68,74 @@ export class ClientErrorSpanProcessor implements SpanProcessor { } } +/** + * Metric export interval in milliseconds. + * + * Deliberately left to OTEL_METRIC_EXPORT_INTERVAL (the SDK's own variable, default 60s) rather + * than pinned in code. An explicit reader takes precedence over the SDK's env handling, so a + * hardcoded value would silently disable that knob — and a shorter interval costs a full + * collect-and-export of *every* instrument, including the auto-instrumentation histograms, on + * the very event loop this is meant to keep free. + */ +export function metricExportIntervalMs(): number | undefined { + const raw = process.env.OTEL_METRIC_EXPORT_INTERVAL; + if (raw == null || raw === '') return undefined; + + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) { + throw new Error(`Invalid OTEL_METRIC_EXPORT_INTERVAL value '${raw}': expected a positive number of milliseconds`); + } + + return parsed; +} + let sdk: NodeSDK | undefined; +/** + * Whether telemetry export is configured at all. Shared with src/runtime-metrics.ts so both + * halves switch on the same condition — the metrics live in the meter provider this module + * registers, and a second copy of the check would drift the moment this one changes. + */ +export function isTelemetryEnabled(): boolean { + return Boolean(process.env.OTEL_EXPORTER_OTLP_ENDPOINT); +} + +/** + * Maps CRON_ROLE to the service name reported with every span, so spans from the worker role are + * distinguishable from the rest instead of arriving under one name. + * + * Reads the environment directly rather than the configuration: startTracing runs before the + * application is created, and importing the configuration here would load the instrumented + * modules before the SDK patches them. The value itself is validated in config.ts. + */ +export function tracingServiceName(): string { + return process.env.CRON_ROLE === 'worker' ? 'dfx-api-worker' : 'dfx-api'; +} + export function startTracing(): NodeSDK | undefined { // Disabled unless a collector endpoint is configured (e.g. on LOC / in tests). - if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT) return undefined; + if (!isTelemetryEnabled()) return undefined; if (sdk) return sdk; + const intervalMs = metricExportIntervalMs(); + sdk = new NodeSDK({ - serviceName: 'dfx-api', + serviceName: tracingServiceName(), // The 4xx-not-a-failure processor runs before the exporting batch // processor so corrected statuses are what gets exported. The exporter // reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment. spanProcessors: [new ClientErrorSpanProcessor(), new BatchSpanProcessor(new OTLPTraceExporter())], + // Metrics travel the same OTLP route as spans, so runtime saturation (see + // src/runtime-metrics.ts) needs no extra endpoint or scrape target. Spans measure how long + // work waited; these measure whether the process had CPU to run it at all. + // + // The reader is declared explicitly because the gauges need a meter provider that is + // guaranteed to exist; the interval stays env-driven so this does not quietly change the + // export cadence the SDK would otherwise use. + metricReader: new PeriodicExportingMetricReader({ + exporter: new OTLPMetricExporter(), + ...(intervalMs == null ? {} : { exportIntervalMillis: intervalMs }), + }), instrumentations: [ getNodeAutoInstrumentations({ // Filesystem spans are pure noise for an API service.