perf: Reduce record batch memory accounting overhead

[ryux1]

Which issue does this PR close?

• Closes Significant overhead in datafusion_common::utils::memory::get_record_batch_memory_size #24310.

Rationale for this change

Record batch memory accounting runs on hot execution paths. The current implementation materializes ArrayData for every array and allocates a hash set even for small batches, adding measurable overhead to queries that frequently update memory reservations.

What changes are included in this PR?

• Traverse Arrow arrays directly and recursively count their backing buffers without materializing ArrayData.
• Track the first 16 buffer identities inline, then promote to a hash set for wider batches or counters spanning many batches.
• Preserve shared-buffer deduplication and full buffer-capacity accounting semantics.
• Add parity coverage against the previous ArrayData traversal for primitive, binary/view, list/view, fixed-size, struct, union, dictionary, map, and run-end encoded layouts.
• Add a focused Criterion benchmark for narrow through wide record batches.

Criterion point estimates from cargo bench -p datafusion-common --bench record_batch_memory, measured in isolated target directories on the same machine:

Columns	main	this PR	Speedup
1	79.327 ns	10.412 ns	7.62x
4	288.97 ns	38.157 ns	7.57x
16	1.1770 us	185.09 ns	6.36x
64	4.9819 us	1.5537 us	3.21x

Are these changes tested?

Yes.

• cargo fmt --all -- --check
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test -p datafusion-common --lib (550 passed)
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
• Focused Criterion comparison shown above

Are there any user-facing changes?

No API or behavior changes. This reduces CPU and allocation overhead in record batch memory accounting.

AI assistance: OpenAI Codex assisted with implementation and test execution. I reviewed the change and its behavior end to end.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.14062% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.15%. Comparing base (bbb552f) to head (e584d8b).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/utils/memory.rs	94.14%	14 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24319    +/-   ##
========================================
  Coverage   81.14%   81.15%            
========================================
  Files        1110     1110            
  Lines      386137   386377   +240     
  Branches   386137   386377   +240     
========================================
+ Hits       313347   313569   +222     
- Misses      54324    54340    +16     
- Partials    18466    18468     +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.