perf(aws-aurora): skip Data API result metadata when column names are statically known#63
Draft
kleberncto wants to merge 1 commit into
Draft
Conversation
… statically known PR sbalmt#61 (released in 0.49.0) swapped formatRecordsAs:JSON for includeResultMetadata:true, expecting to avoid the RDS Data API's per-request catalog introspection (pg_catalog query, ~15-20% of db.load on Gaio's console_db). Production telemetry after the 0.49.0 deploy shows the introspection query unchanged: both formatRecordsAs and includeResultMetadata make the Data API return typed column metadata, which appears to require the same server-side catalog lookup either way. parseFieldRecords only ever used columnMetadata for one thing: the output key per column (label ?? name). Type decoding already works off the Field union's own discriminants, with no columnMetadata involved. And the query builder already knows the output column list for the common ORM-generated shapes - it just wasn't threaded through to the driver. This adds an opportunistic, purely-additive path: - SqlResults#names() (pgsql) walks the same column list getResultColumns builds SQL from, returning the final output keys instead. It returns undefined the moment any column's name isn't statically derivable (bare column reference, unaliased raw column), and an empty list for a result set with no columns (DML without RETURNING - no metadata needed at all in that case). - PgExecuteStatement gains an optional columns list, captured at each prepare* call site in pgclient. It must be captured BEFORE build(): building replaces sub-select column aliases with temporary ones (S0, S1...) while emitting the original alias in the SQL, so reading names after build would yield wrong keys for every select that includes a relation. - The Data API driver only sets includeResultMetadata when statement.columns is absent, and decodes records positionally via the new parseFieldRecordsByNames when it's present. Formatted date/time columns (to_char) already embed `AS <fieldKey>` in the generated expression; the select builder now passes them as raw values carrying that alias (builder.rawValue(generator, alias)) instead of bare column references, so their output name is statically known too - the emitted SQL is byte-identical. Covered: plain selects, selects with relation sub-selects, selects with formatted date/time columns (i.e. full-row selects), count, exists, and DML without RETURNING (zero result columns - metadata skipped entirely). Falls back to today's includeResultMetadata behavior: raw SQL (rawQuery) and insert-with-select (built from column references). Nothing is removed; unknown shapes keep exact current behavior. Validation (no real Aurora cluster available in this environment): - pgsql: full offline suite green (189/189, includes a new result-names spec covering every names() branch and the capture-before-build contract) - pgclient: FULL suite green against a real PostgreSQL 16 (553/553, including every test/client/* round-trip: all scalar/json/relation shapes through the native driver with this diff applied) - pgclient: new test/query/statement-columns spec asserting statement.columns for select (plain, formatted date-time, relation sub-select - the capture-before-build regression), count, exists, update with flag, DML without select (empty list), insert-with-select (undefined -> metadata fallback), and that the emitted SQL keeps to_char + AS unchanged - aws-aurora: typecheck + build clean; smoke assertions for parseFieldRecordsByNames (13/13) Still needed before merging: real-Aurora integration validation (client-*.spec.ts) and confirmation from the maintainer that includeResultMetadata is in fact the trigger for the catalog introspection, not just formatRecordsAs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #61.
Context
After 0.49.0 went live on our production workload (Aurora Serverless v2 + Data API), Performance Insights shows the per-request catalog introspection query (
SELECT c.oid, a.attnum, a.attname ... FROM pg_catalog.pg_class ...) unchanged: ~15% ofdb.loadbefore the deploy vs ~13% after (within traffic noise), still firing at the same rate down to 5-min granularity.The working hypothesis:
formatRecordsAs: JSONandincludeResultMetadata: trueare two ways of asking the Data API for the same typed column metadata, so either one triggers the same server-side catalog lookup. The fields that introspection query selects (attname,relname,nspname,attnotnull,attidentity) map 1:1 to theColumnMetadatashape (name,tableName,schemaName,nullable,isAutoIncrement), and the AWS docs notecolumnMetadatacomes back blank whenformatRecordsAs=JSONis set — two mutually exclusive paths to the same info. If that's right, #61 removed the duplicated JSON parsing on the client but not the introspection cost on the server.I couldn't verify this against a real Aurora cluster from my environment — flagged below as the main thing to confirm.
Solution
parseFieldRecordsonly usescolumnMetadatafor one thing: the output key of each column (label ?? name). Value decoding already works off theFieldunion discriminants alone. And the query builder already knows the output column list for most ORM-generated statements — it just wasn't threaded through to the driver. This PR makes that list part of the statement, so the driver can stop asking the database for result-set metadata whenever the shape is statically known:SqlResults#names()(pgsql): walks the same column listgetResultColumnsbuilds SQL from, returning the final output keys. Returnsundefinedas soon as any column's name isn't statically derivable (bare column reference, raw column without alias), and[]for a result set with no columns (DML withoutRETURNING— no metadata needed at all in that case).PgExecuteStatement.columns(pgclient): optional list captured at eachprepare*call site through a sharedgetStatementColumnshelper. It must be captured beforebuild(): building reassigns sub-select column aliases to temporary ones (S0,S1, ...) while emitting the original alias in the SQL, so reading names after build would yield wrong keys for any select that includes a relation. The helper JSDoc documents this andstatement-columns.spec.tslocks it with a regression test.includeResultMetadatawhenstatement.columnsis absent, and decodes records positionally via the newparseFieldRecordsByNameswhen it's present.to_char(...) AS "field"expression already embeds the output name; the select builder now passes those asbuilder.rawValue(generator, alias)instead of a bare column reference, so their output name is statically known too. The emitted SQL is byte-identical (asserted in the specs).Everything is additive: no emitted SQL changes, and any statement whose output shape isn't fully known keeps exactly today's
includeResultMetadatabehavior.Coverage
Skips result metadata for:
count(__EZ4_COUNT),exists(__EZ4_EXISTS) and the upsert update flag (__EZ4_OK)RETURNING(empty column list — metadata skipped entirely)Falls back to metadata (behavior unchanged):
rawQuery(arbitrary SQL)Validation
result-names.spec.tscovering everynames()branch, including the capture-before-build contracttest/client/*round-trip green with this diff), plus newstatement-columns.spec.tsassertingstatement.columnsper statement shape and that the emitted SQL keepsto_char+ASunchangedGaps / pending confirmation
client-*.spec.ts) — I can't provision a cluster from my environment.includeResultMetadataactually stops thepg_catalogintrospection. The change is structured so correctness holds either way — the pending part is only the perf win.@ez4/transform(using the full table schema) would allow droppingto_charfrom the emitted SQL entirely and covering insert-with-select as well —transformStringwould first need date/date-time/time normalization, which it doesn't do today.