Joe/rust core binding poc#894
Open
joe-clickhouse wants to merge 50 commits into
Open
Conversation
Reconciles the rust codec binding with the pluggable backend architecture from #872. The transform selection hook moves into the new client construction points and the threaded insert path moves into SyncBackendClient.data_insert. The async client hooks survived the merge in place. QBit resolution takes the main fix from #868 wholesale since both fixes are wire identical and the merged transpose functions would otherwise double reverse. Driver internal metadata queries are now marked on the query context and always decode with the Python codec so rust_strict does not fail client initialization. Changelog entries for the released 1.3.0 through 1.5.0 sections are restored to match main with the unreleased rust codec entries moved to the top. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-poc # Conflicts: # CHANGELOG.md # clickhouse_connect/driver/streaming.py # tests/unit_tests/test_streaming_source.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental opt in Rust codec for client managed FORMAT Native traffic. The default codec is unchanged and nothing new is required for existing users.
A new
native_codecclient option selects the codec.pythonis the default and uses the existing Python and Cython implementation.rustprefers the compiled codec and falls back to the Python codec in the rare cases the Rust path does not implement.rust_strictraises instead of routing. The Rust codec covers the entire production type system in both directions, includingNullable,LowCardinality,Array,Tuple,Map,Variant,Dynamic,JSON, geometry types, and supportedAggregateFunctionstates, so fallback will likely be driven by unusual query options such as custom per queryquery_formatsrather than by column types. The option can also be set with a common setting or theCLICKHOUSE_CONNECT_NATIVE_CODECenvironment variable.The codec covers
query,query_np,query_df, their block and row streaming variants, and inserts includinginsert_df, with full production type coverage. The Arrow methods stay on FORMAT Arrow and are unaffected. Driver internal queries always use the Python codec. Fallback decisions happen before any bytes are read or sent, so there is never a mid stream codec switch.Implementation
The codec is a PyO3 binding around ch-core-rs, a shared Rust core that decodes Native bytes into Arrow shaped columnar memory with the GIL released. Decode overlaps with transport per batch. On localhost benchmarks the Rust path wins or ties everywhere, with speedups ranging from roughly 1.3x for Python row results to 7x for NumPy and pandas destinations depending on the workload and types involved.
Packaging
The binding ships as a separate wheel named
clickhouse-connect-core, versioned independently and installed through a newrustextra. The driver checks a binding API version at client creation and raises a clear error naming the install or upgrade command when the module is missing or too old. The core crate is pinned by git tag.Two new workflows:
binding_ci.ymlruns clippy plus the full binding and codec test suites against a live ClickHouse on any PR touching the Rust path, andpublish_core.ymlbuilds the wheel matrix and publishes with PyPI trusted publishing, mirroring the existing publish workflow.Testing
822 binding tests, 362 codec integration tests against live ClickHouse, and handshake unit tests. The full existing suite, mypy, and ruff pass unchanged. A new docs page covers installation, fallback rules, versioning, and the small set of known behavior differences.
Checklist
Delete items not relevant to your PR: