perf(db): cut CPU ~78% via evmlog/output indexes + hot query rewrites

[matheus1lva]

Summary

Neon CPU usage roughly doubled since the start of the year and Neon recently changed CPU billing. pg_stat_statements shows two queries account for 74% of total exec time, seven for ~99%. All are reads against evmlog (5M rows / 5.9 GB) and the output hypertable, and all are caused by missing indexes or missing series_time predicates.

Full breakdown, per-query root cause, and ranked plan: docs/cpu-cost-analysis.md.

Top consumers fixed in this PR

#	%	calls	mean	Root cause	Fix
1	39.4%	1.17 M	9.4 s	projectDebtAllocator full-scans evmlog — PK is (chain_id, address, signature, …) so (chain_id, signature) without address can't use it	Add evmlog(chain_id, signature, block_number DESC, log_index DESC) + partial expr index on (args->>'vault')
2	35.0%	287 k	34 s	fetchStrategyPerformance has no series_time predicate → every Timescale chunk scanned + MAX(block_time) GROUP BY	Rewrite CTE+JOIN as DISTINCT ON (…) … ORDER BY series_time DESC bounded by series_time >= now() - interval '30 days'
3	8.7%	12.7 M	191 ms	Timeseries fanout SELECT DISTINCT block_time over all history per (vault × label)	Query already-bucketed series_time between [start, end]; backed by the new output(chain_id, address, label, series_time DESC) index

Expected combined CPU reduction: ~78% of monthly CPU-seconds.

Schema changes (migration 20260513233437-cpu-indexes)

CREATE INDEX IF NOT EXISTS evmlog_idx_chain_signature
  ON evmlog (chain_id, signature, block_number DESC, log_index DESC);

CREATE INDEX IF NOT EXISTS evmlog_idx_chain_signature_args_vault
  ON evmlog (chain_id, signature, (args->>'vault'))
  WHERE args ? 'vault';

CREATE INDEX IF NOT EXISTS idx_output_chain_address_label_series_time
  ON output (chain_id, address, label, series_time DESC);

Indexes are idempotent (IF NOT EXISTS). Recommended deployment path: create the evmlog indexes CONCURRENTLY against prod first (5.9 GB table — non-concurrent creation will block writes for minutes), then run the migration which becomes a no-op.

Code changes

• packages/ingest/abis/yearn/3/vault/snapshot/hook.ts — fetchStrategyPerformance rewritten (CTE+JOIN → DISTINCT ON + series_time window).
• packages/ingest/fanout/timeseries.ts — DISTINCT block_time → DISTINCT series_time with [start, end] bounds.

Out of scope (follow-ups)

Listed in docs/cpu-cost-analysis.md under the fix plan, in materiality order:

• Cache projectDebtAllocator result in the vault snapshot row.
• Add series_time lower bound to the apr/apy MAX(CASE WHEN …) resolvers (Online ingest loader #4 / Setup yarn workspace #6 / Move .env to root #7).
• Resolver-level cache for timeseries / tvls GQL queries (Online gql #5).
• Kill the event_name COUNT(*) probe in packages/ingest/probe/index.ts (Index vaults and strategies #8 — 47 s per call).
• Tune Neon compute on idle/read replica branches.

Test plan

Setup

• Confirm .env POSTGRES_* points at a dev branch (not prod).
• make dev — starts redis, postgres, ingest, web.

Migration

• Apply migration on the dev DB:

  bun --filter db migrate up
  

  Expected: three CREATE INDEX statements run cleanly; subsequent runs are no-ops.
• Verify indexes exist:

  SELECT indexname FROM pg_indexes
  WHERE tablename IN ('evmlog','output')
    AND indexname IN (
      'evmlog_idx_chain_signature',
      'evmlog_idx_chain_signature_args_vault',
      'idx_output_chain_address_label_series_time'
    );

  Expected: 3 rows.
• Rollback path works:

  bun --filter db migrate down -c 1
  

  Expected: 3 indexes dropped. Then migrate up to re-apply.

Hot query #1 — projectDebtAllocator

• Pick a v3 vault address and run before/after:

  EXPLAIN (ANALYZE, BUFFERS) SELECT args FROM evmlog
  WHERE chain_id = 1
    AND signature = '<NewDebtAllocator topic>'
    AND args->>'vault' = '<vault addr>'
  ORDER BY block_number DESC, log_index DESC LIMIT 1;

  Expected: Index Scan on evmlog_idx_chain_signature_args_vault (or evmlog_idx_chain_signature), execution time < 20 ms (was seconds).
• Terminal UI → ingest → fanout abis and confirm v3 vault snapshots complete without errors.

Hot query #2 — fetchStrategyPerformance

• After a fanout run, spot-check a vault's snapshot via GraphQL and confirm strategy performance fields (oracle.apr, historical.weeklyNet, etc.) populate as before.
• EXPLAIN ANALYZE the rewritten query against the dev DB and confirm it uses idx_output_chain_address_label_series_time with chunk exclusion (Timescale Chunks excluded during runtime).

Hot query #3 — Timeseries fanout

• Terminal UI → ingest → fanout abis (timeseries). Confirm jobs enqueue identically to before for a known vault — same set of endOfDay timestamps queued.
• Specifically test: a vault with a partially-backfilled history should enqueue the missing days only. Compare set of mq.add calls to a baseline if reproducible.

Regression — replays

• fanout replays from terminal UI; confirm replays still complete and produce the same outputs.

Cleanup

• make down

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
kong	Ready	Preview, Comment	May 13, 2026 11:53pm