perf(sqlx): cache struct field metadata to avoid per-row tag parsing - #5731
Open
suhuaqin wants to merge 5 commits into
Open
perf(sqlx): cache struct field metadata to avoid per-row tag parsing#5731suhuaqin wants to merge 5 commits into
suhuaqin wants to merge 5 commits into
Conversation
unmarshalRows re-parsed db tags, re-walked the struct and rebuilt the tagged value map for every row. Cache the type-level mapping (reflect.Type -> field indexes / tag names) in a sync.Map, initialize pointer fields from the cached index list, and match columns once per result set instead of per row. Behavior is preserved: - nil pointer fields are still allocated for every row, including columns that are not selected (previous unwrapFields side effect) - getValueInterface is still the single place producing scan targets, keeping the **T semantics that let database/sql store NULL as nil - strict counting, db:"-" handling, unexported-field errors and the positional untagged path are unchanged (full sqlx test suite passes) Benchmark (23-column struct via sqlmock, 2s benchtime): QueryRowsPartial100: 878486 -> 367485 ns/op (2.39x) QueryRowsPartial10: 171249 -> 147047 ns/op (1.16x) allocs/op: 10978 -> 5479 (2.0x) B/op: 611386 -> 251355 (2.4x)
suhuaqin
force-pushed
the
perf/sqlx-cache-fields
branch
from
August 14, 2026 08:46
35bcfb2 to
8b166ef
Compare
Author
|
Correction to my earlier note above — my previous measurement methodology was flawed (I compared against a stale checkout, so both sides ran the same code). The actual facts, re-verified against unmodified master:
|
added 4 commits
August 14, 2026 17:20
…eparately Address review findings on the strict/named edge cases: - flat (strict count, positional path, pointer init) now follows unwrapFields: unexported fields and db:"-" subtrees are skipped entirely - byName now follows getTaggedFieldValueMap: embedded structs are flattened regardless of their own tag or export status, and db:"-" tags stay in the name map, so all-ignored structs still discard columns silently and inner tagged fields of db:"-" embeddings are still scanned - tagged unexported fields no longer fail at cache build time; they surface ErrNotReadableValue from getValueInterface when a column actually targets them, which also restores the old strict-check-before-error precedence - adds regression tests for all reported shapes plus duplicate-tag resolution
The test relied on scanning 10k mocked rows exceeding a live 2ms deadline. Once scanning got faster (see the sqlx field-cache change) the scan could finish inside the budget and the DeadlineExceeded assertion flipped. Use an already expired deadline instead so the outcome does not depend on scanning speed.
collectByName skipped whole subtrees behind unexported embedded pointers to avoid the nil-intermediate panic, but with the pointer set the old per-row code scanned their inner tagged fields (flagEmbedRO does not propagate into exported inner fields), so those columns were silently discarded. Recurse into such embeddings again and resolve byName paths with a walk that fails with ErrNotReadableValue when crossing a nil embedded pointer ptrIndex cannot pre-allocate (unexported, or db:"-" so collectFlat skipped it) — where the old code panicked. Slice scans always build fresh rows, so only the single-row path can carry a pre-set pointer. Also gofmt orm_bench_test.go.
The old per-row getTaggedFieldValueMap ran getValueInterface on every visited tagged field, allocating its nil pointer as a side effect even when the column was not selected — including db:"-" tagged leaves, tagged fields inside db:"-" or unexported embeddings, and fields whose tag a later duplicate overwrote. The cached path only mirrored the unwrapFields side effect (ptrIndex), so those pointers stayed nil. Collect taggedPtrIndex during the collectByName walk (the walk visits overwritten duplicates too; unexported tagged leaves errored in the old code and still do at scan time, so they stay out), and run it as a second pass in initPtrFields after ptrIndex, via the nil-safe fieldByIndex: a path crossing an unallocated embedded pointer outside ptrIndex is skipped, keeping the ErrNotReadableValue-not-panic posture for columns that actually match. Regression tests cover the db:"-" leaf, the db:"-" embedding subtree (tagged and db:"-" leaves allocated, untagged pointer stays nil) and the overwritten duplicate inside a db:"-" embedding. Bench unchanged within noise (100 rows 430µs before/after).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
unmarshalRows/unmarshalRowre-parsedbstruct tags, re-walk the whole struct and rebuild the tagged value map for every row of a result set. For wide models (20+ columns) under load this reflection work shows up as a significant CPU hotspot in production profiles (we measured ~27% of service CPU in a logistics service dominated by list queries).This PR caches the type-level metadata (
reflect.Type→ flattened field indexes + tag→field mapping) in async.Map, the same approachjmoiron/sqlxuses inStructScan("caches the reflect work of matching columns to struct fields").Key changes in
core/stores/sqlx/orm.go:cachedFields: flattened field indexes, tag-name map, strict count, pointer-field list — built once perreflect.TypeunwrapFieldsallocated every settable nil pointer every row (ptrIndex, even for unselected columns), and the tagged map build additionally allocated every visited tagged field's nil pointer (taggedPtrIndex:db:"-"leaves, tagged fields insidedb:"-"/unexported embeddings, and fields whose tag a later duplicate overwrote)unmarshalRowsmatches column names once per result set instead of once per rowgetValueInterfaceis untouched and remains the single place producing scan targets, keeping the**Tsemantics that letdatabase/sqlstore NULL as nildb:"-"handling, unexported-field errors and the positional untagged path are unchangedRemoved code
getTaggedFieldValueMapandunwrapFieldsare superseded by the cached equivalents (no other callers;getValueInterfaceis kept — it is covered by its own tests and still used for scan targets).Benchmark
New
orm_bench_test.go: 23-column pointer-field struct viasqlmock(mock overhead is identical before/after),go test -bench=BenchmarkQueryRowsPartial -benchmemon an M2, go1.25:(1-row numbers are omitted: at that size the benchmark is dominated by the sqlmock driver itself rather than the scanning code.)
Behavior verification
core/stores/sqlxtest suite passes, includingTestUnmarshalRowsZeroValueStructPtr(NULL → nil pointer semantics) and the strict-mode / embedded-struct cases.Behavior notes (intentional, reviewed)
flatfollows the oldunwrapFields(strict count, positional path, pointer init),byNamefollows the oldgetTaggedFieldValueMap(embeddings flattened regardless of their own tag/export status;db:"-"stays in the name map). Regression tests cover: unexported embedded values in strict mode, all-db:"-"structs silently discarding columns, inner tagged fields ofdb:"-"embeddings, duplicate tags (later wins), and embedded pointers that cannot be pre-allocated (set → scanned as before; nil →ErrNotReadableValue).db:"-"embeddings (whose subtreecollectFlatskips) are not in the pointer-init list, so they can stay nil. byName paths are resolved with a nil-safefieldByIndexwalk instead ofFieldByIndex: a selected column resolving through a nil one fails withErrNotReadableValue— the previous per-row code panicked there (reflect: Field on zero Value) — while a set pointer's inner tagged fields scan exactly as before (reachable on the single-row path, which scans the caller's struct in place; slice scans always build fresh rows).getTaggedFieldValueMaprangetValueInterfaceon every visited tagged field, allocating its nil pointer even when the column was not selected.taggedPtrIndexcollects those fields during thecollectByNamewalk (the walk visits overwritten duplicates too; unexported tagged leaves keep erroring at scan time as before) andinitPtrFieldsruns it as a second pass afterptrIndex, via the nil-safefieldByIndex, skipping paths that cross an unallocated embedded pointer. Regression tests cover thedb:"-"leaf, thedb:"-"embedding subtree (tagged anddb:"-"leaves allocated, untagged pointer stays nil) and the overwritten duplicate inside adb:"-"embedding.ErrNotReadableValuenow surfaces fromgetValueInterfacewhen a column actually targets the field, instead of during the per-row map build. The error type is identical; this also restores the old strict-check-before-error precedence.sync.Mapkeyed byreflect.Type, no eviction (entries bounded by the set of scanned model types), concurrent first builds are idempotent duplicates.TestQueryRowsScanTimeout): its assertion implicitly depended on scanning being slower than a live 2ms budget; this PR's speedup made it flaky. It now uses an already-expired deadline, independent of scanning speed.