Skip to content

perf: optimize non-null FixedSizeBinary grouping - #24285

Open
dhruvxvaishnav wants to merge 1 commit into
apache:mainfrom
dhruvxvaishnav:perf/fixed-size-binary-grouping
Open

perf: optimize non-null FixedSizeBinary grouping#24285
dhruvxvaishnav wants to merge 1 commit into
apache:mainfrom
dhruvxvaishnav:perf/fixed-size-binary-grouping

Conversation

@dhruvxvaishnav

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

FixedSizeBinary grouping keys, such as UUIDs, commonly contain no nulls. The vectorized comparison path currently checks null state and performs bounds checks for every candidate row even when both the stored and incoming values are known to be non-null.

What changes are included in this PR?

  • Add a non-null comparison fast path that skips per-row null checks.
  • Keep checked access in debug builds and use bounds-check-free reads in optimized builds. The group and input row indices are produced internally by GroupValuesColumn and are valid by construction.
  • Preserve the existing nullable comparison path.
  • Add focused coverage for matching values, mismatching values, and rows already rejected by an earlier grouping column.

Are these changes tested?

Yes. The following checks passed:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --release -p datafusion-physical-plan --lib fixed_size_binary (9 passed)
  • RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
  • cargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils -- fixed_size_binary

Two Linux benchmark comparisons against the same unchanged main baseline showed:

Case First run Repeated run
Vectorized, 1,000 groups 11.9% faster 21.4% faster
Vectorized, 1,000,000 groups No statistically significant change 4.0% faster

The row-based control showed no statistically significant change in the full comparison.

Are there any user-facing changes?

There are no API or behavior changes. Non-null FixedSizeBinary group comparisons are faster.

Skip per-row null checks when both stored and incoming FixedSizeBinary values are non-null. Retain checked access in debug builds and use bounds-check-free reads in optimized builds while preserving the nullable fallback.

Add focused coverage for matches, mismatches, and rows rejected by an earlier grouping column.
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 12, 2026
@alamb

alamb commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

run benchmark multi_group_by

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5271374176-1570-4594w 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing perf/fixed-size-binary-grouping (dc75d7d) to 9b3b518 (merge-base) diff

Run configuration
run benchmark multi_group_by

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

Benchmark for this request failed.

Run configuration
run benchmark multi_group_by

Last 20 lines of output:

Click to expand
  Downloaded zerovec v0.11.6
  Downloaded jiff v0.2.24
  Downloaded schemars v1.2.1
  Downloaded zerocopy v0.8.48
  Downloaded hyper v1.9.0
  Downloaded libc v0.2.189
  Downloaded petgraph v0.8.3
  Downloaded rustix v1.1.4
  Downloaded csv v1.4.0
  Downloaded parquet v59.2.0
  Downloaded zstd-sys v2.0.16+zstd.1.5.7
  Downloaded liblzma-sys v0.4.6
  Downloaded ring v0.17.14
  Downloaded sha2-const-stable v0.1.0
  Downloaded linux-raw-sys v0.12.1
  Downloaded aws-lc-sys v0.40.0
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`
error: target `multi_group_by` in package `datafusion-physical-plan` requires the features: `test_utils`
Consider enabling them by passing, e.g., `--features="test_utils"`

File an issue against this benchmark runner

@dhruvxvaishnav

Copy link
Copy Markdown
Author

@alamb The benchmark failed before running because multi_group_by already declares required-features = ["test_utils"], while the benchmark runner invokes Criterion targets with only --features=parquet. This affects both the merge base and this branch, so it is not caused by the PR.

The working command is:

cargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils

No change to this PR seems appropriate. The general runner fix would be to include each target's required-features from cargo metadata when invoking it. Could we rerun after that is supported, or treat the local benchmark results as sufficient?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improved performance for FixedSizeBinary grouping

3 participants