enhancement(clickhouse sink): add Arrow IPC compression

[benjamin-awd]

Summary

Adds a compression option (none | lz4_frame | zstd, default none) to the clickhouse sink's arrow_stream batch encoding. It compresses each column buffer inside the Arrow IPC stream, so ClickHouse decompresses it on ingest.

Today the only way to compress an arrow_stream payload is a whole-payload HTTP gzip Content-Encoding — good on size, heavy on CPU. IPC compression is much cheaper (see issue for benchmark)

Note: Compressed Arrow IPC needs ClickHouse ≥ 23.11 (both lz4 and zstd). Older servers accept the insert but can't read typical small compressed batches (Code: 33 CANNOT_READ_ALL_DATA) and would silently drop data, so the sink checks the server version at startup and rejects compression on older servers.

Default is none, so existing configs are unaffected.

Vector configuration

sinks:
  my_clickhouse:
    type: clickhouse
    inputs: [my_source]
    endpoint: http://clickhouse:8123
    database: mydb
    table: mytable
    format: arrow_stream
    batch_encoding:
      codec: arrow_stream
      compression: zstd   # none (default) | lz4_frame | zstd

How did you test this PR?

• Unit tests (lib/codecs): compressed streams round-trip back to the same RecordBatch and are smaller than uncompressed; version parsing and the minimum-version check.
• Integration tests against real ClickHouse containers:
  • insert_events_arrow_format_{zstd,lz4}_compression — insert and read back; skip below 23.11.
  • arrow_compression_rejected_on_old_clickhouse — the sink refuses to build with compression on ClickHouse < 23.11.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Add block-level compression for ClickHouse sink (ArrowStream) #25756
• Related: Add ArrowStream format to Clickhouse sink #24074, enhancement(clickhouse sink): Add ArrowStream format #24373

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

[benjamin-awd]

Took this implementation from the Datadog Zerobus sink to make it easier to consolidate in a follow-up PR

vector/src/sinks/databricks_zerobus/config.rs

Lines 53 to 70 in 0f7f98c

	pub enum Compression {
	/// No compression.
	#[default]
	None,
	/// LZ4 frame compression.
	Lz4Frame,
	/// Zstandard compression.
	Zstd,
	}
	impl From<Compression> for Option<arrow::ipc::CompressionType> {
	fn from(value: Compression) -> Self {
	match value {
	Compression::None => None,
	Compression::Lz4Frame => Some(arrow::ipc::CompressionType::LZ4_FRAME),
	Compression::Zstd => Some(arrow::ipc::CompressionType::ZSTD),
	}
	}

Moving it here instead of re-defining it in the Clickhouse sink since anything that implements Arrow Stream IPC will likely want to use these compression options

cc @flaviofcruz

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a07062340

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".