Non-Custodial Partner Program — aggregated partner statistics endpoints by joshuakrueger-dfx · Pull Request #4538 · DFXswiss/api

joshuakrueger-dfx · 2026-07-31T07:38:51Z

Why

Not symptom-driven: no incident triggered this — a wallet partner asked for usage numbers, and
the only route that answers today hands out per-transaction detail.
Scale: Cake alone has 126,988 users in production and is the first of several wallet partners;
/v2/kyc/client/payments serves the same CLIENT_COMPANY role up to 1000 individual transactions
per call, including customer IBANs, with free date filters.
Smaller fix considered: point partners at that existing endpoint and let them aggregate —
insufficient because it makes every partner store per-transaction data about their customers,
including users who never consented to it, purely to compute counts and volumes they can be handed
directly.

Wallet partners integrating DFX have no way to see how their integration performs. The only
partner-facing transaction endpoint returns per-transaction detail — more than a usage dashboard
needs, and more than a partner should hold. This is the interface half of the Non-Custodial Partner
Program; #4508 is the lock that must ship first.

Generic by design: there is no partner-specific code anywhere. Any wallet gets its own numbers
through the same route.

What

GET /v1/statistic/partner and GET /v1/statistic/partner/timeline, scoped to the wallet in the
company JWT. There is deliberately no walletId parameter — a partner cannot express a request
for someone else's data. Both return aggregates only: volume and transaction counts by direction,
active and new users, breakdowns by asset, fiat currency, blockchain and payment method, plus the
referral position in EUR.

What the partner gets, and what stays behind

The point of these endpoints is scope, not secrecy. The data itself stays retrievable — DFX keeps
full access, and GET /v2/kyc/client/payments already serves per-transaction detail to the same
CLIENT_COMPANY role, up to 1000 rows per call with free date filters. What this PR adds is a route
that answers the dashboard question — volume, counts, users, breakdowns — without the partner
holding per-transaction data at all. A partner integrating against these two endpoints stores
aggregates, not customer transactions.

That framing matters for the paragraphs below: the k-anonymity threshold here reduces what a
low-privilege integration sees. It is not the system's privacy boundary, because the same
credential can reach the detail endpoint directly. Whether that endpoint should stay as open as it
is, is a separate question and deliberately out of scope here.

Anonymity: what it protects, and what it does not

Aggregates alone are not anonymous. Three independent reviews demonstrated, with executed output,
that a suppressed value could be recovered by subtraction. What the endpoints do about it:

Block suppression on the period totals. If any direction falls below the threshold, the whole
group is suppressed — members, total and derived average. No per-cell nulling, because
total − visible would recover the hidden member exactly.
Timeline thresholds per direction, not on the bucket sum. A bucket of {buy: 5, sell: 1}
previously passed and disclosed the single sell in the clear.
Periods snap to UTC day boundaries with a minimum span of one day, so two windows differing by
hours cannot isolate a single transaction.
The threshold binds to distinct users as well as transactions. Five trades by one person used
to pass every gate.
Query budget per wallet. The shared RateLimitGuard keys by IP prefix and bypasses known and
Azure addresses, so it puts no bound on repeated scrapes from a single partner JWT. The new guard
keys by the wallet id in the JWT, has no bypass list, and fails closed if the wallet id is absent:
120 requests per hour per wallet on each route.

Five reconstruction paths remain open. This is a decision, not an oversight — each was measured
against the alternative, and against the fact that the same caller can read the underlying
transactions directly. Closing them would cost visibility a partner legitimately needs, to withhold
numbers that are one endpoint away:

Period differencing. [Jan 1, Jan 31) and [Jan 1, Jan 30) are both far above the threshold
and both are served. Their difference is Jan 31 exactly, including a day with a single
transaction that the timeline would have withheld. Volume and transaction counts are additive so
the subtraction is exact. activeUsers overlaps across periods and is not — but newUsers counts
user.created in the window and is additive, so it differences exactly, down to a single
registration.
Breakdown rows against the totals. Rows under the threshold are removed from the breakdown
while the direction totals stay visible, and both come from the same query. totals − visible rows returns the removed row. Closing this means removing further rows until the withheld mass
itself clears the threshold, which costs visibility in exactly the breakdown a partner reads.
Running counters. allTime, referral and registeredUsers are not period-bound. Two polls
seconds apart expose a single trade as the delta, for any partner above the threshold. Closing
this means serving them day-aligned, which makes a dashboard's headline number up to 24 hours
stale.
Granularity differencing. Complementary suppression picks the smallest filled bucket of the
whole response, which for a day-granularity answer usually lands in a different week than the
withheld day. Asking the same period again at granularity=Week returns that week visibly, and
week total − visible days of that week is the withheld day. Nothing couples the two answers,
and constraining that would mean holding query state across requests.
Breakdown against block-suppressed totals — inside a single response. Totals suppression and
breakdown suppression run independently. fiatCurrencies, blockchains and paymentMethods are
merged across directions, while assets carries a direction field. With one direction below
the threshold the totals group is nulled, but the merged rows stay visible, and
Σ merged − Σ assets of the other directions is the withheld direction's volume — no second
request needed. Closing it means coupling the two suppression stages, which would drop the
breakdown for every partner whose smallest direction is thin.

The budget slows all three; it does not close them. It is also inactive unless
REQUEST_LIMIT_CHECK=true is set, and counts in memory per instance — so the effective ceiling is
120/h times the number of app instances. Stated plainly here because the budget is part of the
argument and should not be read as stronger than it is.

Verified by executed output rather than argument: with sell below the threshold,
total − buy − swap yields null, not the hidden value.

Correctness fixes found in review

referral.creditOpen used partnerRefCredit − paidRefCredit. paidRefCredit covers both pots;
the three sibling sites use refCredit + partnerRefCredit − paidRefCredit. The old formula
understated the figure and could go negative.
Timeline bucketing mixed UTC period snapping with local date-part keys. Off UTC this shifted every
bucket by a day. Truncation is now bound to UTC in SQL and the bucket keys follow.
map.set on bucket keys silently dropped a bucket on collision; it accumulates now.
Date ranges are half-open, so consecutive windows no longer double-count a boundary transaction.

Second review round

Four unreachable branches removed. An isNaN check behind a parseDate that already throws;
a spanDays < 1 rejection that the day snap plus the from >= to check already make impossible;
a || 1 fallback whose condition cannot hold (isUnderThreshold is false for 0, so a suppressed
group always has at least one non-zero member); and a duplicated loop that wrote the same values
twice, together with a thinking-out-loud comment that had no business shipping.
meta.suppressedBuckets → meta.suppressedCount. On the timeline the field really did count
buckets; on the summary it counts suppressed fields and breakdown rows, where there are no buckets
at all. Both routes are new and unconsumed, so the name is corrected before anyone depends on it.
The coverage pin moved to the load-bearing files. Only partner-statistic.enum.ts was pinned —
42 lines of constants — while the suppression module and the rate-limit guard were unguarded.
Both now sit at 100% on all four metrics and are pinned; the 942-line service is deliberately not,
but the gaps that mattered are closed.

Verification

Full suite on 8f3847010, rebased onto current develop: 371 suites (362 passed, 9 skipped),
7068 tests (6861 passed, 207 skipped), 0 failures. lint empty, format:check clean, tsc --noEmit clean, build
clean, coverage gate green. Measured off-CI, on a machine with the repo's own scripts — this
revision has not been through CI yet. Six of the 8 skipped suites are the real-Postgres specs,
which skip wherever MIGRATION_TEST_PG is unset; CI sets it on every shard, so those do run there.
The other two are unrelated: one is a hard describe.skip, the other needs SEPOLIA_RPC_URL,
which CI does not set. The partner integration spec was additionally run against a real
Postgres 17 with TZ=UTC on this revision: 5 of 5 green. CodeQL flagged three critical
type-confusion alerts on the date parser — Express hands over an array when a query parameter is
repeated and the regex guard coerced it silently, so both parsers now reject non-string input
before any pattern matching.
Statistic specs run green under four process timezones — UTC, Europe/Zurich, Pacific/Auckland,
America/Los_Angeles. Before the fix, Europe/Zurich produced 4 buckets where 3 were expected.
Mutation probes, not just deletion probes. Two of them previously proved the tests measured
nothing: neutralising mergeNamedRows failed 0 of 32, and dropping the wallet scope from a
single query failed 0 of 18 because the scope assertion used some instead of every. The
some→every change did not actually close the second one — the assertion filtered the clauses
by walletId before checking them, so a removed clause vanished from the list and the check passed
vacuously for four of the six user-touching queries. Removing the scope from any one of the six now
fails 2 of 51, measured per query, and substitution is covered separately. Both are red now, alongside probes for block suppression, day snapping, the distinct-user threshold,
the creditOpen formula and the per-direction timeline threshold. Second round, each probe
anchored to a uniquely matched line and reverted afterwards: bucket-collision accumulation
+= → = (1 of 26 red), the same mutation in mergeNamedRows (1 of 26), ISO-week
alignment day === 0 ? 6 : day - 1 → day - 1 (1 of 26), the bucket person-floor
Math.max → Math.min (1 of 29), and the guard's fail-closed throw replaced by the old IP
fallback (1 of 5). The timeline-collision path and both week/month alignment branches had no
test at all before this round, although the collision fix was already claimed above.
Third round — the suppression itself was untested in the response path. Five mutations that
removed k-anonymity from the delivered payload each failed 0 of 26: neutralising the breakdown
suppression, neutralising the timeline suppression, passing the internal users field through in
either payload, and serving newUsers unsuppressed. The suite stayed green while the endpoints
would have shipped every withheld number and the per-row person counts. Each of those five now
fails 1 of 43. Strict date parsing is covered the same way: reverting to the tolerant
new Date(value) fails 3 of 43, exactly the three rejected forms.
The real-Postgres integration spec calls the actual service methods — getStatistics,
getTimeline, aggregateByDirection — against real rows, following the repo's existing
MIGRATION_TEST_PG convention. It previously re-implemented the query builders by hand, so a
broken join in the real chain would have left it green while it claimed "real Postgres".

For reviewers

TZ was pinned nowhere in the repo; this PR sets ENV TZ=UTC in the runtime image and logs the
detected zone at start. It only warns — the deploy start command lives outside this repo and may
set TZ itself, so a hard abort would risk the deploy for a premise nothing here can verify. Two
existing modules build bucket keys from local date parts on the assumption that "the app and DB
both run UTC"; they are worth a separate look.
No index is added, and that is a decision with a number behind it. The new aggregates read
buy_crypto (130,180 rows) and buy_fiat (68,862 rows) — queried against production on
2026-07-31, not estimated. At that size a filtered aggregate is milliseconds; an index for a
dashboard call that runs twice an hour would be cost without a case. If either table grows by an
order of magnitude, the question is worth reopening.
Wallet scope lives in SQL for the aggregates, which is new for this kind of query. Scoping a
single-row lookup in SQL already exists (kyc.service.ts filters wallet: { id: walletId }); what
is new is doing it inside aggregate queries over the transaction tables. The partner endpoints that
read many rows load the wallet with its users and pass the ids on (kyc-client.service.ts,
transaction.service.ts batching at 100). Measured against production, that does not carry an aggregate endpoint: Cake has
126,988 users, so the existing pattern would mean 1,270 sequential batches per query and roughly
22,800 round trips per summary call, plus every user row in memory. The SQL scope keeps it at 18
queries and loads no user rows. Every user-touching query carries the clause, and a test now fails
when any one of the six loses it — removal, not just substitution.
One call fans out to concurrent queries. Concurrency is bounded by a per-request semaphore
every SQL execution passes through, so nested fan-outs share one budget of 4. The previous
per-call-site cap did not compose: four breakdown tasks each opened their own pool of workers and
a single request reached 11 simultaneous queries against a default pool of 10.
Suppression protects against reconstruction across day-aligned windows. It is not a formal
query-budget guarantee, and path 5 above means it does not hold within a single response either.

Wallet partners integrating DFX have no way to see how their integration performs. The only partner-facing transaction endpoint returns per-transaction detail including customer IBANs, which is both more than a usage dashboard needs and more than a partner should hold. Add GET /statistic/partner and /statistic/partner/timeline, scoped to the wallet in the company JWT. There is deliberately no walletId parameter, so a partner cannot express a request for someone else's data. Both endpoints return aggregates only: volume and transaction counts by direction, active and new users, breakdowns by asset, fiat currency, blockchain and payment method, and the referral position in EUR. Aggregates alone are not anonymous. Counts below a threshold of five are suppressed; the effective count is min(transactions, distinct users). Additive groups (period totals by direction, timeline buckets) use block suppression — if any member is under the threshold the whole group is null — so a hidden value cannot be recovered via totals − visible. Period bounds snap to UTC day boundaries with a half-open interval and a one-day minimum. Referral open credit uses refCredit + partnerRefCredit − paidRefCredit on the wallet owner account; tradingUsers and referral balances are gated by the same threshold. Rate limits for these routes count by wallet id, not IP prefix.

…overage

…ery concurrency Three independent reviews found the k-anonymity wiring untested: neutralising the breakdown or timeline suppression, or passing the internal users field through to the payload, failed 0 of 26 tests. Those five mutations now fail 1 of 43 each. - Conformity: PascalCase enum values with explicit maps for JSON field names and the DATE_TRUNC unit, plain Error instead of InternalServerErrorException, and BuyCryptoRepository no longer re-provided next to its exporting module. - Dead code: the uncalled amlPassOnly option, three unused suppression helpers and the comments describing a settlement stage that does not exist. - Concurrency: a per-request semaphore every SQL execution passes through, so nested fan-outs share one budget instead of one cap per call site. - Dates: default period computed in pure UTC over 30 inclusive days, strict ISO parsing that rejects offsetless timestamps, NaN aggregates and a missing wallet row now throw instead of surfacing as null or zero.

Columns such as `created` are `timestamp without time zone`. The Postgres driver serializes JS Date values in process-local wall time and Postgres drops the offset, so a non-UTC process stores shifted values and the partner-statistic day buckets drift. Nothing pinned the timezone: it held only because node:20-alpine has no /etc/localtime and therefore defaults to UTC. ENV TZ=UTC in the runtime image makes that explicit, and the boot check fails closed in deployed environments before TypeORM connects. LOC only warns, so developers outside UTC can still start the app.

…troller The integration spec ran its SQL through the real service methods for the first time and exposed that its own harness was never built for concurrent connections: `SET search_path` is per connection, the fan-out takes up to four, and three of four tests failed against real Postgres with "relation does not exist". The DataSource now carries `schema`, which keeps the concurrency the test is meant to exercise. It additionally requires a UTC process timezone and skips loudly otherwise, rather than failing on a shifted period bound. - `parseDate` rejected offsetless timestamps but still accepted 2024-06-31 and rolled it into July. - The controller had no spec at all: guard order, jwt.user as walletId, the throttle budget and the granularity default were wired but never verified. - `granularity` typed as the enum, `maxActive` and two stale comments removed.

…borting on a non-UTC process The tenant isolation was only covered incidentally. The scope assertion filtered the where-clauses by `walletId` before checking them, so a removed clause vanished from the list and the check passed vacuously; removal was caught only because the fixture-routed value assertions happened to notice. The check now pins the expected clause count with an anchored pattern, so removing, falsifying or widening the condition fails directly — an `OR 1 = 1` appended to the scope no longer passes. The same shape covered the AML filter and the period bounds, both unpinned until now. In the controller spec, guard slot 0 was never asserted: substituting AuthGuard on both routes kept all ten tests green. The timezone check sampled the current offset, so a process on Europe/London passed all winter and would have written shifted timestamps from late March, while a summer restart of the same deployment would have refused to boot. It now samples January and July, and it only warns: the deploy start command lives outside this repo and may set TZ itself, so a hard abort would risk the deploy for a premise nothing here can verify. Both the passing and the failing case are logged, so the next deploy shows what the process actually sees. `bootstrap()` now has a catch, so a boot-time config error surfaces as itself instead of through the uncaught-exception handler. Also documents why the wallet scope sits in SQL rather than following the load-wallet-and- pass-ids pattern the row-reading partner endpoints use: at 126,988 Cake users that pattern means ~1,270 sequential batches per query.

Three assertions had the same vacuum-true shape as the wallet scope: they filtered the collected clauses first and then checked the survivors, so a missing condition simply left the list. The AML filter could be made conditional and the SELL period bound replaced by a tautology with every test green, which would have counted rejected AML traffic in partner volume and turned period-scoped SELL volume into all-time. Two of the three carried a comment claiming the opposite. All three now pin the expected count against an anchored pattern, so removal, falsification and widening each fail. The integration spec still sampled the timezone offset once, the defect fixed in the previous commit two files over. It now uses the shared year-round check. One spec was order-dependent and failed roughly once in eighteen runs: the harness let an unscoped fixture merge drop the flag the missing-wallet case asserts on. Twenty consecutive runs are green, and a regression test pins the harness behaviour itself.

…red list `length > 0` on a filtered list is blind to a missing element: dropping one of the three DATE_TRUNC group-bys left the timeline assertion green. Pinning the count fails it, and the integration spec now pins the single expected BTC breakdown row rather than accepting any number of them.

Five assertions were blind to a condition being added rather than removed or altered. Adding a group-by on the transaction id to the timeline query collapses COUNT(DISTINCT user.id) to one per bucket, pushing every bucket under the threshold, and no test noticed; the breakdown group-bys were blind in both directions; and a fourth, unscoped UNION leg in the active-user count passed every pinned count while reading across all wallets. The pins now cover the whole list rather than a filtered subset, and the UNION legs are counted, not just their clauses — the clause total alone would still have missed a leg carrying no WHERE. One assertion in the suppression spec sat inside a condition that never holds, so the privacy claim in its name was never checked. `jest.useFakeTimers()` was only undone on the success path, so one failing assertion left the timers stopped and the next test died in a five-second timeout — a real failure manufacturing a second, unrelated one. An `afterEach` now restores them either way. This also corrects the previous commit: the cause it states for the intermittent failure does not hold. The path it describes is never reached from the service, and the failure does not reproduce on the pre-fix harness either. The added harness test is sound but inert, the timer leak cannot explain a solitary failure, and the actual cause is still unknown.

… depending on connection state The pinned clause counts proved that the known queries carry their conditions, not that the query set is unchanged. Switching the active-user UNION to UNION ALL passed every assertion while double-counting anyone who both buys and sells — and that count is the gate the whole suppression layer hangs on, so a three-person cohort would report six and clear the threshold of five. It is now asserted against real Postgres with a user active in two directions. The period bounds were pinned as clause text only, so widening the bound value rather than the clause slipped through; the bound parameters are asserted now. Two pins were brittle enough to invite deletion: the group-by list was order-pinned, so a harmless reorder of the fan-out turned it red, and four magic numbers were duplicated across three tests. The list is compared as a multiset and the numbers live in one place. The integration setup created its tables unqualified and relied on a session-scoped `SET search_path` while the fan-out hands out up to four pooled connections, so the following test could land on a backend that never saw the schema. Fully qualified names plus a single connection for the setup; fifty consecutive runs are green. The failing assertion behind the earlier intermittent failure was still never captured, so this closes a demonstrated defect, not a proven cause.

…sed base The clause assertion introduced with the previous round filtered the collected clauses by `created` and then checked the survivors, so dropping a period bound from one query left the list without failing anything — the same shape that round was written to remove. The count is pinned now. The rebase onto develop merged both sides' coverage pins; the doc counts are recomputed from the config rather than carried over from either side.

CodeQL flagged three critical type-confusion alerts: the date parser is typed for `string | Date`, but Express hands over a `string[]` when a query parameter appears twice, and the regex guard in front of it does not catch that — `RegExp.test` coerces the array to a string, so a single-element array passes it. The slices that follow then run on an array and produce something else entirely. It ended in a 400 only because `Number` of an array is `NaN` and the calendar comparison failed, which is luck rather than a check. Both parsers now reject anything that is neither a string nor a Date before any pattern matching, and say so in the message. Four tests cover single- and multi-element arrays; without the guards they fail with the old accidental message instead of the typed one.

… coverage The coverage ratchet reported them as complete but unpinned — a green check carrying an annotation, which is how an unguarded file quietly erodes later. Both are pinned now, the controller as logic and the DTO as declarative, and the doc counts are recomputed from the config rather than incremented by hand.

joshuakrueger-dfx · 2026-08-01T22:07:58Z

Superseded by #4587 — same tree, history cleaned up (fourteen review-round commits collapsed onto two). Closing this one.

joshuakrueger-dfx mentioned this pull request Jul 31, 2026

Non-Custodial Partner Program — partner statistic completion funnels #4539

Closed

joshuakrueger-dfx marked this pull request as ready for review July 31, 2026 07:53

joshuakrueger-dfx requested review from TaprootFreak and davidleomay as code owners July 31, 2026 07:53

joshuakrueger-dfx marked this pull request as draft July 31, 2026 10:14

joshuakrueger-dfx added 12 commits August 1, 2026 22:21

Pin partner statistic enum in coverage gate

dc79e78

Fail closed on missing wallet and pin partner statistic suppression c…

d47d582

…overage

joshuakrueger-dfx force-pushed the feat/partner-statistics branch from 5a188c3 to 49894e7 Compare August 1, 2026 20:36

github-advanced-security AI found potential problems Aug 1, 2026

View reviewed changes

Comment thread src/subdomains/core/statistic/partner-statistic.service.ts Fixed

Comment thread src/subdomains/core/statistic/partner-statistic.service.ts Fixed

Comment thread src/subdomains/core/statistic/partner-statistic.service.ts Fixed

joshuakrueger-dfx added 2 commits August 1, 2026 23:10

joshuakrueger-dfx marked this pull request as ready for review August 1, 2026 21:56

joshuakrueger-dfx mentioned this pull request Aug 1, 2026

Non-Custodial Partner Program — aggregated partner statistics endpoints #4587

Open

joshuakrueger-dfx closed this Aug 1, 2026

joshuakrueger-dfx deleted the feat/partner-statistics branch August 1, 2026 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Non-Custodial Partner Program — aggregated partner statistics endpoints - #4538

Non-Custodial Partner Program — aggregated partner statistics endpoints#4538
joshuakrueger-dfx wants to merge 14 commits into
developfrom
feat/partner-statistics

joshuakrueger-dfx commented Jul 31, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuakrueger-dfx commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshuakrueger-dfx commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

What the partner gets, and what stays behind

Anonymity: what it protects, and what it does not

Correctness fixes found in review

Second review round

Verification

For reviewers

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuakrueger-dfx commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshuakrueger-dfx commented Jul 31, 2026 •

edited

Loading