perf: optimize non-null FixedSizeBinary grouping - #24285
Conversation
Skip per-row null checks when both stored and incoming FixedSizeBinary values are non-null. Retain checked access in debug builds and use bounds-check-free reads in optimized builds while preserving the nullable fallback. Add focused coverage for matches, mismatches, and rows rejected by an earlier grouping column.
|
run benchmark multi_group_by |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/fixed-size-binary-grouping (dc75d7d) to 9b3b518 (merge-base) diff Run configurationrun benchmark multi_group_byResults will be posted here when complete File an issue against this benchmark runner |
|
Benchmark for this request failed. Run configurationrun benchmark multi_group_byLast 20 lines of output: Click to expandFile an issue against this benchmark runner |
|
@alamb The benchmark failed before running because The working command is:
No change to this PR seems appropriate. The general runner fix would be to include each target's |
Which issue does this PR close?
Rationale for this change
FixedSizeBinary grouping keys, such as UUIDs, commonly contain no nulls. The vectorized comparison path currently checks null state and performs bounds checks for every candidate row even when both the stored and incoming values are known to be non-null.
What changes are included in this PR?
GroupValuesColumnand are valid by construction.Are these changes tested?
Yes. The following checks passed:
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo test --release -p datafusion-physical-plan --lib fixed_size_binary(9 passed)RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryptioncargo bench -p datafusion-physical-plan --bench multi_group_by --features test_utils -- fixed_size_binaryTwo Linux benchmark comparisons against the same unchanged
mainbaseline showed:The row-based control showed no statistically significant change in the full comparison.
Are there any user-facing changes?
There are no API or behavior changes. Non-null FixedSizeBinary group comparisons are faster.