Skip to content

perf(aws-aurora): skip Data API result metadata when column names are statically known#63

Draft
kleberncto wants to merge 1 commit into
sbalmt:mainfrom
kleberncto:perf/aurora-data-api-static-column-names
Draft

perf(aws-aurora): skip Data API result metadata when column names are statically known#63
kleberncto wants to merge 1 commit into
sbalmt:mainfrom
kleberncto:perf/aurora-data-api-static-column-names

Conversation

@kleberncto

Copy link
Copy Markdown
Contributor

Follow-up to #61.

Context

After 0.49.0 went live on our production workload (Aurora Serverless v2 + Data API), Performance Insights shows the per-request catalog introspection query (SELECT c.oid, a.attnum, a.attname ... FROM pg_catalog.pg_class ...) unchanged: ~15% of db.load before the deploy vs ~13% after (within traffic noise), still firing at the same rate down to 5-min granularity.

The working hypothesis: formatRecordsAs: JSON and includeResultMetadata: true are two ways of asking the Data API for the same typed column metadata, so either one triggers the same server-side catalog lookup. The fields that introspection query selects (attname, relname, nspname, attnotnull, attidentity) map 1:1 to the ColumnMetadata shape (name, tableName, schemaName, nullable, isAutoIncrement), and the AWS docs note columnMetadata comes back blank when formatRecordsAs=JSON is set — two mutually exclusive paths to the same info. If that's right, #61 removed the duplicated JSON parsing on the client but not the introspection cost on the server.

I couldn't verify this against a real Aurora cluster from my environment — flagged below as the main thing to confirm.

Solution

parseFieldRecords only uses columnMetadata for one thing: the output key of each column (label ?? name). Value decoding already works off the Field union discriminants alone. And the query builder already knows the output column list for most ORM-generated statements — it just wasn't threaded through to the driver. This PR makes that list part of the statement, so the driver can stop asking the database for result-set metadata whenever the shape is statically known:

  • SqlResults#names() (pgsql): walks the same column list getResultColumns builds SQL from, returning the final output keys. Returns undefined as soon as any column's name isn't statically derivable (bare column reference, raw column without alias), and [] for a result set with no columns (DML without RETURNING — no metadata needed at all in that case).
  • PgExecuteStatement.columns (pgclient): optional list captured at each prepare* call site through a shared getStatementColumns helper. It must be captured before build(): building reassigns sub-select column aliases to temporary ones (S0, S1, ...) while emitting the original alias in the SQL, so reading names after build would yield wrong keys for any select that includes a relation. The helper JSDoc documents this and statement-columns.spec.ts locks it with a regression test.
  • Data API driver (aws-aurora): only sets includeResultMetadata when statement.columns is absent, and decodes records positionally via the new parseFieldRecordsByNames when it's present.
  • Formatted date/time columns: the to_char(...) AS "field" expression already embeds the output name; the select builder now passes those as builder.rawValue(generator, alias) instead of a bare column reference, so their output name is statically known too. The emitted SQL is byte-identical (asserted in the specs).

Everything is additive: no emitted SQL changes, and any statement whose output shape isn't fully known keeps exactly today's includeResultMetadata behavior.

Coverage

Skips result metadata for:

  • plain selects and selects with relation sub-selects
  • selects with formatted date/time columns (i.e. full-row selects)
  • count (__EZ4_COUNT), exists (__EZ4_EXISTS) and the upsert update flag (__EZ4_OK)
  • DML without RETURNING (empty column list — metadata skipped entirely)

Falls back to metadata (behavior unchanged):

  • rawQuery (arbitrary SQL)
  • insert-with-select (its returning wrap is built from column references whose output names aren't derivable — possible follow-up)

Validation

  • pgsql: 189/189 — new result-names.spec.ts covering every names() branch, including the capture-before-build contract
  • pgclient: full suite 553/553 against a real PostgreSQL 16 (every test/client/* round-trip green with this diff), plus new statement-columns.spec.ts asserting statement.columns per statement shape and that the emitted SQL keeps to_char + AS unchanged
  • aws-aurora: typecheck + build clean; positional decode covered by targeted assertions
  • eslint clean on all three packages

Gaps / pending confirmation

  1. Real Aurora integration run (client-*.spec.ts) — I can't provision a cluster from my environment.
  2. The hypothesis itself: confirming on a real cluster (or via Performance Insights) that dropping includeResultMetadata actually stops the pg_catalog introspection. The change is structured so correctness holds either way — the pending part is only the perf win.
  3. Separate follow-up idea: schema-driven response normalization with @ez4/transform (using the full table schema) would allow dropping to_char from the emitted SQL entirely and covering insert-with-select as well — transformString would first need date/date-time/time normalization, which it doesn't do today.

… statically known

PR sbalmt#61 (released in 0.49.0) swapped formatRecordsAs:JSON for
includeResultMetadata:true, expecting to avoid the RDS Data API's
per-request catalog introspection (pg_catalog query, ~15-20% of db.load
on Gaio's console_db). Production telemetry after the 0.49.0 deploy
shows the introspection query unchanged: both formatRecordsAs and
includeResultMetadata make the Data API return typed column metadata,
which appears to require the same server-side catalog lookup either way.

parseFieldRecords only ever used columnMetadata for one thing: the
output key per column (label ?? name). Type decoding already works
off the Field union's own discriminants, with no columnMetadata
involved. And the query builder already knows the output column list
for the common ORM-generated shapes - it just wasn't threaded through
to the driver.

This adds an opportunistic, purely-additive path:

- SqlResults#names() (pgsql) walks the same column list getResultColumns
  builds SQL from, returning the final output keys instead. It returns
  undefined the moment any column's name isn't statically derivable
  (bare column reference, unaliased raw column), and an empty list for
  a result set with no columns (DML without RETURNING - no metadata
  needed at all in that case).
- PgExecuteStatement gains an optional columns list, captured at each
  prepare* call site in pgclient. It must be captured BEFORE build():
  building replaces sub-select column aliases with temporary ones
  (S0, S1...) while emitting the original alias in the SQL, so reading
  names after build would yield wrong keys for every select that
  includes a relation.
- The Data API driver only sets includeResultMetadata when
  statement.columns is absent, and decodes records positionally via
  the new parseFieldRecordsByNames when it's present.

Formatted date/time columns (to_char) already embed `AS <fieldKey>`
in the generated expression; the select builder now passes them as
raw values carrying that alias (builder.rawValue(generator, alias))
instead of bare column references, so their output name is statically
known too - the emitted SQL is byte-identical.

Covered: plain selects, selects with relation sub-selects, selects
with formatted date/time columns (i.e. full-row selects), count,
exists, and DML without RETURNING (zero result columns - metadata
skipped entirely). Falls back to today's includeResultMetadata
behavior: raw SQL (rawQuery) and insert-with-select (built from
column references). Nothing is removed; unknown shapes keep exact
current behavior.

Validation (no real Aurora cluster available in this environment):
- pgsql: full offline suite green (189/189, includes a new
  result-names spec covering every names() branch and the
  capture-before-build contract)
- pgclient: FULL suite green against a real PostgreSQL 16 (553/553,
  including every test/client/* round-trip: all scalar/json/relation
  shapes through the native driver with this diff applied)
- pgclient: new test/query/statement-columns spec asserting
  statement.columns for select (plain, formatted date-time, relation
  sub-select - the capture-before-build regression), count, exists,
  update with flag, DML without select (empty list), insert-with-select
  (undefined -> metadata fallback), and that the emitted SQL keeps
  to_char + AS unchanged
- aws-aurora: typecheck + build clean; smoke assertions for
  parseFieldRecordsByNames (13/13)

Still needed before merging: real-Aurora integration validation
(client-*.spec.ts) and confirmation from the maintainer that
includeResultMetadata is in fact the trigger for the catalog
introspection, not just formatRecordsAs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant