Release: develop -> main

docs/coverage-gate.md

@@ -6,7 +6,7 @@ the other.
 | Gate             | Config                         | Scope                                      | Question it answers                                      |
 | ---------------- | ------------------------------ | ------------------------------------------ | -------------------------------------------------------- |
 | Frick gate       | `jest.frick.config.js`         | 10 Frick files, run by 10 Frick specs only | Do _these specs alone_ fully cover _these files_?        |
-| Coverage ratchet | `jest.coverage-gate.config.js` | 421 files, whole suite                     | Has coverage regressed anywhere it was already complete? |
+| Coverage ratchet | `jest.coverage-gate.config.js` | 422 files, whole suite                     | Has coverage regressed anywhere it was already complete? |
 
 ## What the ratchet is, and what it is not
 
@@ -16,7 +16,7 @@ file, CI fails.
 
 It is a **regression gate**, not a statement about test quality:
 
-- It does not claim the repo is well tested. Overall coverage is 59.57% of statements and 42.46%
+- It does not claim the repo is well tested. Overall coverage is 59.61% of statements and 42.64%
   of branches; the pinned files are the subset that happens to be complete today.
 - It does not verify that a file's _own_ spec covers it. Under a whole-suite run, coverage may
   come from any spec. The Frick gate is the one that makes the stronger per-spec claim, which is
@@ -25,7 +25,7 @@ It is a **regression gate**, not a statement about test quality:
   ratchet only protects files already on the list, and that list grows by hand (see "How the list
   grows"). That is the price of the threshold approach.
 
-Of the 421 pinned files, **232 carry real logic** (they have functions and/or branches) and
+Of the 422 pinned files, **233 carry real logic** (they have functions and/or branches) and
 **189 are purely declarative today** (NestJS modules, constant files with neither). The two groups
 are kept visibly separate in the config so the count is not mistaken for test depth.
 
@@ -150,7 +150,7 @@ warm caches, is a good deal slower than the 1.5 min it takes in CI.
 
 ## Current state
 
-Measured on develop @ 045e6f8d6 with this PR's tests applied.
+Measured on develop @ 77a106207 with this PR's change applied.
 
 The collection glob matches 1,657 files under `src/`. 1,606 of them contain instrumentable code
 and appear in the report. The remaining 51 compile to no executable statements and therefore
@@ -161,11 +161,11 @@ deleting them would be a separate cleanup.
 
 | Class    | Files | Meaning                                         |
 | -------- | ----- | ----------------------------------------------- |
-| Complete | 421   | Pinned by the ratchet                           |
-| Partial  | 1,058 | Some coverage, below 100 on at least one metric |
+| Complete | 422   | Pinned by the ratchet                           |
+| Partial  | 1,057 | Some coverage, below 100 on at least one metric |
 | None     | 127   | No coverage at all                              |
 
-Totals: statements 59.57%, branches 42.46%, functions 34.16%, lines 59.93%.
+Totals: statements 59.61%, branches 42.64%, functions 34.21%, lines 59.97%.
 
 Coverage is very unevenly distributed. `subdomains/supporting/payout` has 69 of 102 files
 complete; `subdomains/supporting/dex` has 6 of 170, `subdomains/supporting/payin` 6 of 102, and
@@ -175,12 +175,12 @@ six under `subdomains/generic/admin` have no coverage at all.
 ## How the list grows
 
 Any PR may add files to `coverageThreshold` once they reach 100%.
-`jest.coverage-gate.config.js` holds the 421 paths in two arrays, `PINNED_LOGIC` (logic-carrying
+`jest.coverage-gate.config.js` holds the 422 paths in two arrays, `PINNED_LOGIC` (logic-carrying
 files) and `PINNED_DECLARATIVE` (purely declarative files), from which `coverageThreshold` is
 generated. Adding a file means appending its path to the matching array, not writing out a
 `coverageThreshold` object entry by hand.
 
-The intended next step is the set already within reach: **26 files sit at ≥90% on all four
+The intended next step is the set already within reach: **25 files sit at ≥90% on all four
 metrics**, several of them one or two uncovered branches away. Examples:
 
 | File                                                                         | branches | functions | lines | statements |
@@ -196,15 +196,15 @@ always returns at least one element) and `+(raw.legCount ?? 0)` over a SQL `COUN
 never null. Covering such a branch would require inventing a mock the data source cannot
 actually produce, which proves nothing. The correct fix is to remove the unreachable fallback,
 which is also what the project's rule against silent fallbacks calls for.
-`ledger-mark-to-market.service.ts` sits at 92.3% branches for exactly this reason.
+Both examples above were closed that way: the fallback was deleted, not covered.
 
 To regenerate the full picture, run the gate and read `coverage-gate/coverage-summary.json`.
 
 **Removing a file from the list is not a normal fix.** If a change makes a pinned file drop
 below 100, the expected response is to extend the tests. Unpinning is an explicit decision that
 belongs in the PR description, not a silent edit.
 
-That rule stays hard for the 232 logic-carrying files. A foreseeable friction case is different:
+That rule stays hard for the 233 logic-carrying files. A foreseeable friction case is different:
 when one of the 189 purely declarative files (a NestJS module, a constants file) first gains
 executable logic — for example a `useFactory` on a module — the function metric jumps from 0/0 to
 0/N and the gate turns red. Tests remain the preferred fix, but unpinning that one file is an

jest.coverage-gate.config.js

@@ -84,6 +84,7 @@ const PINNED_LOGIC = [
   'src/subdomains/core/accounting/services/ledger-account.service.ts',
   'src/subdomains/core/accounting/services/ledger-booking-job.service.ts',
   'src/subdomains/core/accounting/services/ledger-bootstrap.service.ts',
+  'src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts',
   'src/subdomains/core/aml/enums/aml-list-status.enum.ts',
   'src/subdomains/core/aml/enums/aml-reason.enum.ts',
   'src/subdomains/core/aml/enums/aml-rule.enum.ts',

src/shared/services/process.service.ts

@@ -22,6 +22,7 @@ export enum Process {
   LIQUIDITY_MANAGEMENT_CHECK_BALANCES = 'LiquidityManagementCheckBalances',
   MONITORING = 'Monitoring',
   MONITOR_CONNECTION_POOL = 'MonitorConnectionPool',
+  MONITOR_EVENT_LOOP = 'MonitorEventLoop',
   UPDATE_STATISTIC = 'UpdateStatistic',
   KYC = 'Kyc',
   KYC_IDENT_REVIEW = 'KycIdentReview',

src/shared/utils/__tests__/queue-handler.spec.ts

@@ -0,0 +1,58 @@
+import { QueueHandler } from 'src/shared/utils/queue-handler';
+
+describe('QueueHandler', () => {
+  it('runs queued items and returns their results', async () => {
+    const queue = new QueueHandler(1000, undefined, 1);
+
+    await expect(queue.handle(async () => 42)).resolves.toBe(42);
+
+    queue.stop();
+  });
+
+  it('does not execute items whose queue timeout fired while they were still waiting', async () => {
+    const queue = new QueueHandler(100, undefined, 1);
+    const ran: number[] = [];
+
+    const first = queue.handle(async () => {
+      ran.push(1);
+      await new Promise((resolve) => setTimeout(resolve, 250));
+    });
+    const second = queue.handle(async () => {
+      ran.push(2);
+    });
+
+    await expect(first).rejects.toThrow('Queue timeout');
+    await expect(second).rejects.toThrow('Queue timeout');
+
+    // let the first action finish and the queue drain — the second must have been discarded
+    await new Promise((resolve) => setTimeout(resolve, 300));
+    expect(ran).toEqual([1]);
+
+    queue.stop();
+  });
+
+  it('frees the worker slot via item timeout when an action never settles', async () => {
+    const queue = new QueueHandler(undefined, 50, 1);
+
+    const hanging = queue.handle(() => new Promise(() => undefined));
+    await expect(hanging).rejects.toThrow();
+
+    // slot must be free again: a follow-up item still runs
+    await expect(queue.handle(async () => 'ok')).resolves.toBe('ok');
+
+    queue.stop();
+  });
+
+  it('rejects when the action throws synchronously and frees the worker slot for the next item', async () => {
+    const queue = new QueueHandler(1000, undefined, 1);
+
+    const throwing = queue.handle((): Promise<number> => {
+      throw new Error('boom');
+    });
+    await expect(throwing).rejects.toThrow('boom');
+
+    await expect(queue.handle(async () => 'ok')).resolves.toBe('ok');
+
+    queue.stop();
+  });
+});

src/shared/utils/queue-handler.ts

@@ -7,29 +7,40 @@ class QueueItem<T> {
   private resolve: (value: T | PromiseLike<T>) => void;
   private reject: (e: Error) => void;
 
+  private settled = false;
+
   constructor(
     private readonly action: () => Promise<T>,
     timeout?: number,
   ) {
     this.promise = new Promise((resolve, reject) => {
       this.resolve = (v) => {
+        this.settled = true;
         if (this.timeout) clearTimeout(this.timeout);
         resolve(v);
       };
       this.reject = (e) => {
+        this.settled = true;
         if (this.timeout) clearTimeout(this.timeout);
         reject(e);
       };
     });
     if (timeout) this.timeout = setTimeout(() => this.reject(new Error('Queue timeout')), timeout);
   }
 
+  get isSettled(): boolean {
+    return this.settled;
+  }
+
   public wait(): Promise<T> {
     return this.promise;
   }
 
   public async doWork(timeout: number) {
-    const promise = timeout ? Util.timeout(this.action(), timeout) : this.action();
+    // Defer the call so a synchronous throw inside the action becomes a rejection
+    // instead of escaping doWork and leaving the item unsettled forever.
+    const action = Promise.resolve().then(() => this.action());
+    const promise = timeout ? Util.timeout(action, timeout) : action;
     await promise.then(this.resolve).catch(this.reject);
   }
 
@@ -81,7 +92,11 @@ export class QueueHandler {
     while (this.isRunning) {
       try {
         if (this.queue.length > 0 && this.workParallelCounter < this.maxWorkParallel) {
-          const work = this.queue.shift().doWork(this.itemTimeout);
+          const item = this.queue.shift();
+          // already settled (queue timeout while waiting): the caller is gone, don't run the action
+          if (item.isSettled) continue;
+
+          const work = item.doWork(this.itemTimeout);
 
           this.workParallelCounter++;
           void work.finally(() => this.workParallelCounter--);

src/subdomains/core/accounting/services/ledger-mark-to-market.service.ts

@@ -183,13 +183,13 @@ export class LedgerMarkToMarketService {
       .getRawOne<{
         native: string | null;
         baseUnits: string | null;
-        legCount: string | null;
-        valuedCount: string | null;
+        legCount: string;
+        valuedCount: string;
         decimals: number | null;
         chf: string | null;
       }>();
 
-    const allValued = raw?.baseUnits != null && +(raw.legCount ?? 0) === +(raw.valuedCount ?? 0);
+    const allValued = raw?.baseUnits != null && +raw.legCount === +raw.valuedCount;
     const nativeBalance =
       raw?.decimals != null && allValued
         ? Util.round(Number(BigInt(raw.baseUnits as string)) / 10 ** raw.decimals, 8)

src/subdomains/core/monitoring/monitor-event-loop.service.ts

@@ -0,0 +1,35 @@
+import { Injectable, OnModuleDestroy } from '@nestjs/common';
+import { CronExpression } from '@nestjs/schedule';
+import { monitorEventLoopDelay } from 'perf_hooks';
+import { DfxLogger } from 'src/shared/services/dfx-logger';
+import { Process } from 'src/shared/services/process.service';
+import { DfxCron } from 'src/shared/utils/cron';
+
+@Injectable()
+export class MonitorEventLoopService implements OnModuleDestroy {
+  private readonly logger = new DfxLogger(MonitorEventLoopService);
+
+  private readonly histogram = monitorEventLoopDelay({ resolution: 20 });
+
+  constructor() {
+    this.histogram.enable();
+  }
+
+  // Disable the histogram so its sampling timer does not continue after module teardown.
+  onModuleDestroy(): void {
+    this.histogram.disable();
+  }
+
+  @DfxCron(CronExpression.EVERY_10_SECONDS, { process: Process.MONITOR_EVENT_LOOP })
+  monitorEventLoop(): void {
+    const toMs = (ns: number) => Math.round(ns / 1e6);
+
+    this.logger.info(
+      `EventLoop delay: mean ${toMs(this.histogram.mean)}ms / p95 ${toMs(
+        this.histogram.percentile(95),
+      )}ms / max ${toMs(this.histogram.max)}ms`,
+    );
+
+    this.histogram.reset();
+  }
+}

src/subdomains/core/monitoring/monitoring.module.ts

@@ -14,6 +14,7 @@ import { FiatPayInModule } from 'src/subdomains/supporting/fiat-payin/fiat-payin
 import { NotificationModule } from 'src/subdomains/supporting/notification/notification.module';
 import { PricingModule } from 'src/subdomains/supporting/pricing/pricing.module';
 import { MonitorConnectionPoolService } from './monitor-connection-pool.service';
+import { MonitorEventLoopService } from './monitor-event-loop.service';
 import { HealthController } from './health.controller';
 import { MonitoringController } from './monitoring.controller';
 import { MonitoringService } from './monitoring.service';
@@ -53,6 +54,7 @@ import { SystemStateSnapshotRepository } from './system-state-snapshot.repositor
     SystemStateSnapshotRepository,
     MonitoringService,
     MonitorConnectionPoolService,
+    MonitorEventLoopService,
     NodeBalanceObserver,
     NodeHealthObserver,
     PaymentObserver,