Skip to content

perf(sqlx): cache struct field metadata to avoid per-row tag parsing - #5731

Open
suhuaqin wants to merge 5 commits into
zeromicro:masterfrom
suhuaqin:perf/sqlx-cache-fields
Open

perf(sqlx): cache struct field metadata to avoid per-row tag parsing#5731
suhuaqin wants to merge 5 commits into
zeromicro:masterfrom
suhuaqin:perf/sqlx-cache-fields

Conversation

@suhuaqin

@suhuaqin suhuaqin commented Aug 14, 2026

Copy link
Copy Markdown

What does this PR do?

unmarshalRows / unmarshalRow re-parse db struct tags, re-walk the whole struct and rebuild the tagged value map for every row of a result set. For wide models (20+ columns) under load this reflection work shows up as a significant CPU hotspot in production profiles (we measured ~27% of service CPU in a logistics service dominated by list queries).

This PR caches the type-level metadata (reflect.Type → flattened field indexes + tag→field mapping) in a sync.Map, the same approach jmoiron/sqlx uses in StructScan ("caches the reflect work of matching columns to struct fields").

Key changes in core/stores/sqlx/orm.go:

  • cachedFields: flattened field indexes, tag-name map, strict count, pointer-field list — built once per reflect.Type
  • pointer fields are initialized from cached index lists, preserving both previous side effects: unwrapFields allocated every settable nil pointer every row (ptrIndex, even for unselected columns), and the tagged map build additionally allocated every visited tagged field's nil pointer (taggedPtrIndex: db:"-" leaves, tagged fields inside db:"-"/unexported embeddings, and fields whose tag a later duplicate overwrote)
  • unmarshalRows matches column names once per result set instead of once per row
  • getValueInterface is untouched and remains the single place producing scan targets, keeping the **T semantics that let database/sql store NULL as nil
  • strict counting, db:"-" handling, unexported-field errors and the positional untagged path are unchanged

Removed code

  • getTaggedFieldValueMap and unwrapFields are superseded by the cached equivalents (no other callers; getValueInterface is kept — it is covered by its own tests and still used for scan targets).

Benchmark

New orm_bench_test.go: 23-column pointer-field struct via sqlmock (mock overhead is identical before/after), go test -bench=BenchmarkQueryRowsPartial -benchmem on an M2, go1.25:

Rows ns/op before ns/op after speedup allocs/op before → after B/op before → after
10 171249 147047 1.16x 1161 → 612 65894 → 29986
100 878486 367485 2.39x 10978 → 5479 611386 → 251355

(1-row numbers are omitted: at that size the benchmark is dominated by the sqlmock driver itself rather than the scanning code.)

Behavior verification

  • Full core/stores/sqlx test suite passes, including TestUnmarshalRowsZeroValueStructPtr (NULL → nil pointer semantics) and the strict-mode / embedded-struct cases.

Behavior notes (intentional, reviewed)

  • Semantics preserved via two mirrored collections: flat follows the old unwrapFields (strict count, positional path, pointer init), byName follows the old getTaggedFieldValueMap (embeddings flattened regardless of their own tag/export status; db:"-" stays in the name map). Regression tests cover: unexported embedded values in strict mode, all-db:"-" structs silently discarding columns, inner tagged fields of db:"-" embeddings, duplicate tags (later wins), and embedded pointers that cannot be pre-allocated (set → scanned as before; nil → ErrNotReadableValue).
  • Embedded pointers that cannot be pre-allocated: unexported embeddings (reflect cannot set them) and db:"-" embeddings (whose subtree collectFlat skips) are not in the pointer-init list, so they can stay nil. byName paths are resolved with a nil-safe fieldByIndex walk instead of FieldByIndex: a selected column resolving through a nil one fails with ErrNotReadableValue — the previous per-row code panicked there (reflect: Field on zero Value) — while a set pointer's inner tagged fields scan exactly as before (reachable on the single-row path, which scans the caller's struct in place; slice scans always build fresh rows).
  • Tagged-pointer map-build side effect preserved: the per-row getTaggedFieldValueMap ran getValueInterface on every visited tagged field, allocating its nil pointer even when the column was not selected. taggedPtrIndex collects those fields during the collectByName walk (the walk visits overwritten duplicates too; unexported tagged leaves keep erroring at scan time as before) and initPtrFields runs it as a second pass after ptrIndex, via the nil-safe fieldByIndex, skipping paths that cross an unallocated embedded pointer. Regression tests cover the db:"-" leaf, the db:"-" embedding subtree (tagged and db:"-" leaves allocated, untagged pointer stays nil) and the overwritten duplicate inside a db:"-" embedding.
  • Error timing for tagged unexported fields: ErrNotReadableValue now surfaces from getValueInterface when a column actually targets the field, instead of during the per-row map build. The error type is identical; this also restores the old strict-check-before-error precedence.
  • Cache design: global sync.Map keyed by reflect.Type, no eviction (entries bounded by the set of scanned model types), concurrent first builds are idempotent duplicates.
  • Test fix included (TestQueryRowsScanTimeout): its assertion implicitly depended on scanning being slower than a live 2ms budget; this PR's speedup made it flaky. It now uses an already-expired deadline, independent of scanning speed.

⚠️ No breaking change: all public APIs and scan semantics are unchanged.

unmarshalRows re-parsed db tags, re-walked the struct and rebuilt the
tagged value map for every row. Cache the type-level mapping
(reflect.Type -> field indexes / tag names) in a sync.Map, initialize
pointer fields from the cached index list, and match columns once per
result set instead of per row.

Behavior is preserved:
- nil pointer fields are still allocated for every row, including
  columns that are not selected (previous unwrapFields side effect)
- getValueInterface is still the single place producing scan targets,
  keeping the **T semantics that let database/sql store NULL as nil
- strict counting, db:"-" handling, unexported-field errors and the
  positional untagged path are unchanged (full sqlx test suite passes)

Benchmark (23-column struct via sqlmock, 2s benchtime):
  QueryRowsPartial100: 878486 -> 367485 ns/op (2.39x)
  QueryRowsPartial10:  171249 -> 147047 ns/op (1.16x)
  allocs/op:           10978 -> 5479 (2.0x)
  B/op:                611386 -> 251355 (2.4x)
@suhuaqin
suhuaqin force-pushed the perf/sqlx-cache-fields branch from 35bcfb2 to 8b166ef Compare August 14, 2026 08:46
@suhuaqin

suhuaqin commented Aug 14, 2026

Copy link
Copy Markdown
Author

Correction to my earlier note above — my previous measurement methodology was flawed (I compared against a stale checkout, so both sides ran the same code). The actual facts, re-verified against unmodified master:

  • TestQueryRowsScanTimeout passes deterministically on master (3/3 full-suite runs on the same machine).
  • It starts failing intermittently with this PR, because the cached-field scanning is fast enough to occasionally finish 10k mocked rows inside the live 2ms deadline, flipping the DeadlineExceeded assertion. The test's correctness implicitly depended on the implementation being slow.
  • Fixed in 1d8dc8a by using an already-expired deadline, making the assertion independent of scanning speed. Apologies for the confusion in the earlier comment.

苏华钦 added 4 commits August 14, 2026 17:20
…eparately

Address review findings on the strict/named edge cases:

- flat (strict count, positional path, pointer init) now follows unwrapFields:
  unexported fields and db:"-" subtrees are skipped entirely
- byName now follows getTaggedFieldValueMap: embedded structs are flattened
  regardless of their own tag or export status, and db:"-" tags stay in the
  name map, so all-ignored structs still discard columns silently and inner
  tagged fields of db:"-" embeddings are still scanned
- tagged unexported fields no longer fail at cache build time; they surface
  ErrNotReadableValue from getValueInterface when a column actually targets
  them, which also restores the old strict-check-before-error precedence
- adds regression tests for all reported shapes plus duplicate-tag resolution
The test relied on scanning 10k mocked rows exceeding a live 2ms deadline.
Once scanning got faster (see the sqlx field-cache change) the scan could
finish inside the budget and the DeadlineExceeded assertion flipped.
Use an already expired deadline instead so the outcome does not depend
on scanning speed.
collectByName skipped whole subtrees behind unexported embedded pointers
to avoid the nil-intermediate panic, but with the pointer set the old
per-row code scanned their inner tagged fields (flagEmbedRO does not
propagate into exported inner fields), so those columns were silently
discarded. Recurse into such embeddings again and resolve byName paths
with a walk that fails with ErrNotReadableValue when crossing a nil
embedded pointer ptrIndex cannot pre-allocate (unexported, or db:"-"
so collectFlat skipped it) — where the old code panicked. Slice scans
always build fresh rows, so only the single-row path can carry a
pre-set pointer.

Also gofmt orm_bench_test.go.
The old per-row getTaggedFieldValueMap ran getValueInterface on every
visited tagged field, allocating its nil pointer as a side effect even
when the column was not selected — including db:"-" tagged leaves,
tagged fields inside db:"-" or unexported embeddings, and fields whose
tag a later duplicate overwrote. The cached path only mirrored the
unwrapFields side effect (ptrIndex), so those pointers stayed nil.

Collect taggedPtrIndex during the collectByName walk (the walk visits
overwritten duplicates too; unexported tagged leaves errored in the old
code and still do at scan time, so they stay out), and run it as a
second pass in initPtrFields after ptrIndex, via the nil-safe
fieldByIndex: a path crossing an unallocated embedded pointer outside
ptrIndex is skipped, keeping the ErrNotReadableValue-not-panic posture
for columns that actually match.

Regression tests cover the db:"-" leaf, the db:"-" embedding subtree
(tagged and db:"-" leaves allocated, untagged pointer stays nil) and
the overwritten duplicate inside a db:"-" embedding. Bench unchanged
within noise (100 rows 430µs before/after).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant