Skip to content

perf(polars): rewrite DataFrame serializer with typed column iterators - #30

Merged
idclark merged 4 commits into
mainfrom
perf/dataframe-serializer
Jul 15, 2026
Merged

perf(polars): rewrite DataFrame serializer with typed column iterators#30
idclark merged 4 commits into
mainfrom
perf/dataframe-serializer

Conversation

@idclark

@idclark idclark commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Closes #29.

The polars DataFrame → line protocol serializer accessed data row-by-row via col.get(row_idx) (per-cell AnyValue dispatch) and allocated an intermediate String per field via format!, giving it the opposite performance character of the Point path. This PR rewrites it on top of the Point path's machinery, in two commits:

  1. refactor(point): extract the ryu/itoa/bool/string value writers from FieldValue::write_lp into shared pub(crate) functions. Pure refactor.
  2. perf(polars): rewrite dataframe_to_line_protocol:
    • Columns are resolved, downcast to typed ChunkedArray iterators, and name-escaped once before the row loop; iterators advance in lockstep per row (chunk boundaries handled natively — covered by a new test).
    • Values write through the shared ryu/itoa writers into a single output buffer; all-null-field rows are dropped by truncating back to the row start. No more per-line Strings or final join.
    • Exotic dtypes (i128, binary, list, …) keep a per-row AnyValue fallback with unchanged stringification semantics.
    • Timestamp unit conversion is baked into the reader at construction; written via itoa.

Benchmark

New criterion bench (cargo bench --features polars), 10k rows × 25 columns (1 string tag, 20×f64, 4×i64, datetime timestamp) — the wide-numeric parquet-backfill shape from the user report:

time throughput
before 25.9 ms 387K rows/s
after 10.2 ms 976K rows/s

~2.5× faster (criterion: −60% ± 2%, p < 0.05). Remaining time is dominated by ryu float formatting, which is inherent to the output.

Semantics

Public API unchanged; all pre-existing tests pass unmodified except to_timestamp's unit test, which was rewritten to pin the same dtype/precision matrix against the new timestamp_reader (the old function became dead code). Two tests added: narrow/unsigned dtype suffixes, and multi-chunk columns.

One deliberate, visible change: floats now format via ryu on the DataFrame path too (matching Point), so extreme magnitudes emit exponent notation (1e30) instead of expanded decimals. Line protocol accepts both.

🤖 Generated with Claude Code

idclark and others added 3 commits July 14, 2026 12:28
Pull the ryu/itoa/bool/string field writers out of FieldValue::write_lp
into pub(crate) functions so the DataFrame serializer can reuse them and
the two paths cannot diverge on formatting rules. No behavior change.

Refs #29

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace per-cell AnyValue access and format! allocations with typed
ChunkedArray iterators advanced in lockstep, writing through the shared
ryu/itoa value writers into a single output buffer:

- Columns are resolved, downcast, and name-escaped once before the row
  loop; exotic dtypes keep the AnyValue fallback with unchanged
  stringification semantics.
- All-null-field rows are dropped by truncating the buffer; per-line
  Strings and the final join are gone.
- Timestamps use itoa with unit conversion baked into the reader.

Floats now format via ryu on both paths (matching Point), so extreme
magnitudes emit exponent notation instead of expanded decimals.

Criterion benchmark (10k rows x 25 columns, wide numeric telemetry):
25.9ms -> 10.2ms, ~2.5x faster (387K -> 976K rows/s).

Closes #29

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.87786% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.63%. Comparing base (4db8ce6) to head (e797557).

Files with missing lines Patch % Lines
src/write_dataframe.rs 84.23% 35 Missing ⚠️
src/point.rs 95.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
- Coverage   83.44%   82.63%   -0.81%     
==========================================
  Files          10       10              
  Lines        1836     1981     +145     
==========================================
+ Hits         1532     1637     +105     
- Misses        304      344      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@idclark
idclark requested a review from alespour July 14, 2026 17:40

@alespour alespour left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice improvements, looks all good to me 🚀

@idclark
idclark merged commit 27c0973 into main Jul 15, 2026
9 checks passed
@idclark
idclark deleted the perf/dataframe-serializer branch July 15, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFrame serializer is row-oriented and allocation-heavy, unlike the Point path

3 participants