Execution rule for Chunked<ParquetVariant>#8039
Conversation
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Merging this PR will degrade performance by 16.59%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | chunked_varbinview_opt_canonical_into[(1000, 10)] |
187.7 µs | 225.1 µs | -16.59% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing adamg/chunked-parquet-variant (133ec1d) with develop (008c1d9)
|
@joseph-isaacs has done quite a bit of work on Chunked execution. Maybe we should do the swizzling in the builder execution mode? |
joseph-isaacs
left a comment
joseph-isaacs
left a comment
There was a problem hiding this comment.
did you verify this actually gets hit?
it looks like the right idea
| }; | ||
|
|
||
| let typed_value = if let Some(typed_value_dtype) = layout.typed_value_dtype { | ||
| let typed_value_chunks = chunks |
There was a problem hiding this comment.
does this not get pulled out into the canonical array variant on construction
3 participants
Summary
When we have Chunked array, we can (sometimes) keep it as a ParquetVariant array instead of just shifting the ChunkedArray into a canonical Variant array's core storage child.