Skip to content

d393cad0 - Aggregate the finance change log in SQL instead of loading the month's entities - #4535

Draft
TaprootFreak wants to merge 1 commit into
developfrom
fix/log-job-sql-aggregation
Draft

d393cad0 - Aggregate the finance change log in SQL instead of loading the month's entities#4535
TaprootFreak wants to merge 1 commit into
developfrom
fix/log-job-sql-aggregation

Conversation

@TaprootFreak

Copy link
Copy Markdown
Collaborator

Problem

LogJobService.saveTradingLog() runs every minute. Inside it, getChangeLog() loaded every
buy-crypto, buy-fiat, exchange and payout row of the current month as fully hydrated entities —
only to reduce them to a handful of fee sums.

Because of TypeORM eager relations, one buy-crypto row expands to 481 columns across 15 LEFT
JOINs
. Measured against production data on the 30th of the month:

rows loaded 3 225
result set 9.0 MB, ~1.55 M field values
server-side execution 58 ms
duration the database logged for the same statement 7 926 ms

The gap is not query time — it is the Node process deserialising rows. A CPU profile of the running
API attributed 36.1 % of wall time to the garbage collector and 17.9 % to parseRow in the
pg driver, with application code below 1 %. Event-loop utilisation sat at a median of 96 % with a
p90 delay above 1 s, which is what made unrelated endpoints (asset and fiat lists, buy/paymentInfos)
take hundreds of milliseconds.

Change

The four sources now aggregate in SQL and return only the totals:

before after
BuyCryptoService.getBuyCrypto(from, relations) getBuyCryptoFee(from){ regular, paymentLink }
BuyFiatService.getBuyFiat(from, relations) getBuyFiatFee(from){ regular, paymentLink }
ExchangeTxService.getExchangeTx(from) getExchangeTxFee(from) → one row per exchange/type
PayoutService.getPayoutOrders(from) getPayoutOrderFee(from) → one row per context

The removed methods had no other callers. getFeeAmount() in LogJobService becomes obsolete with
them. The shape of the FinancialChangesLog entry is unchanged.

Semantics were preserved deliberately, not approximated:

  • cryptoInput is LEFT-joined, so buy-crypto rows without a crypto input (bank purchases — the
    majority) still count as regular, exactly as the previous !p.cryptoInput?.paymentLinkPayment
    filter did.
  • The payout fee is COALESCE(prep, 0) + COALESCE(payout, 0), mirroring the entity getter, where a
    NULL column contributed 0 via JS null + x === x. Two such rows exist in the current month; a plain
    prep + payout would have turned them into NULL and swallowed the fee that is set.
  • SUM over an empty set is NULL and is mapped to 0 — no rows in the period means no fees.

Verification

Beyond the unit tests, the SQL that TypeORM generates for all four aggregates was run read-only
against the production database and compared to the FinancialChangesLog entry the current
implementation had just written:

figure new aggregate value written by the running job
buyCrypto 97 501.48000000001 97 501.4799999999
buyFiat 7 771.17 7 771.169999999999
paymentLink 20.341779999999996 20.341779999999993
binance trading 1 300.6799999999994 1 300.6799999999985
scrypt trading 1 990.22 1 990.2199999999998
mexc trading 68.37999999999998 68.38000000000001
payout (non-ref) 242.86 242.85999999999643
payout (ref) 3.400000000000001 3.399999999999992

The remaining differences are float summation order, in the 11th significant digit.

The same four statements execute in 15–33 ms each.

Tests

  • Four new *.pg.spec.ts suites run the aggregates against Postgres semantics via pg-mem, following
    the existing trading-rule.service.pg.spec.ts pattern — a mocked repository executes no SQL, so a
    wrong CASE, a missing GROUP BY, a dropped COALESCE or a flipped date comparison would pass unnoticed.
    Covered: the regular/payment-link split, rows without a crypto input, negative rebates netting
    within a group, NULL fees, the date boundary and the empty period.
  • log-job.service.spec.ts keeps covering how getChangeLog assembles the totals, now against the
    aggregate shapes, plus two new cases: the buy-fee split across both sources and the separation of
    the ref payout context from all others.

…s entities

getChangeLog runs every minute and loaded every buy-crypto, buy-fiat, exchange and
payout row of the current month as fully hydrated entities, only to reduce them to a
few fee sums. Eager relations expand a single buy-crypto row to 481 columns across 15
LEFT JOINs: 3225 rows amount to 9 MB and ~1.55 M field values per run. The database
answers that in 58 ms; deserialising it in the pg driver took up to 7926 ms and kept
the event loop saturated.

Replace the four loaders with SQL aggregates that return only the totals. Semantics
are preserved: cryptoInput stays LEFT-joined so rows without one keep counting as
regular fees, the payout fee COALESCEs both columns the way the entity getter relied
on JS null coercion, and an empty period maps NULL to 0.

Cover the aggregates with pg-mem suites, since a mocked repository executes no SQL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant