chore: remove cross-implementation benchmarks and rewrite README

.github/actions/setup-rust/action.yml

@@ -1,5 +1,5 @@
 name: "Setup Rust"
-description: "Toolchain setup and Initial compilation"
+description: "Toolchain setup and cache configuration"
 
 inputs:
   targets:
@@ -15,13 +15,11 @@ runs:
       run: echo "version=$(cat rust-toolchain.toml | grep channel | awk -F'\"' '{print $2}')" >> $GITHUB_OUTPUT
 
     - name: Rust Toolchain
-      id: rust-toolchain
       uses: dtolnay/rust-toolchain@master
-      if: steps.rustup-cache.outputs.cache-hit != 'true'
       with:
         toolchain: "${{ steps.rust-version.outputs.version }}"
         targets: "${{inputs.targets || ''}}"
-        components: clippy, rustfmt
+        components: rust-src, clippy, rustfmt
 
     - name: Rust Dependency Cache
       uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2

.gitignore

@@ -46,9 +46,3 @@ perf.data.old
 # Local Claude / agent state
 .claude/settings.local.json
 
-# Python / uv
-.venv/
-__pycache__/
-*.pyc
-*.egg-info/
-

CHANGELOG.md

@@ -5,6 +5,34 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased](https://github.com/spiraldb/onpair/compare/v0.1.1...HEAD)
+
+### Added
+
+- Make `CompactDictionary` storage-backed, allowing validated dictionary bytes
+  and offsets to be borrowed or shared without copying.
+- Separate dictionary safety validation from correctness validation, allowing
+  bounded decoding and tokenization checks without requiring full semantic
+  validation.
+
+### Removed
+
+- Remove the obsolete cross-implementation benchmark harness and standalone
+  TPC-H example, retaining the Rust benchmarks under `benches/`.
+
+### Fixed
+
+- Ensure the Rust setup action installs the pinned toolchain without referring to a nonexistent cache step.
+
+## [0.1.1](https://github.com/spiraldb/onpair/compare/v0.1.0...v0.1.1) - 2026-07-15
+
+### Fixed
+
+- Reject compact dictionaries containing more than 65,536 tokens, which cannot
+  be addressed by the `u16` token type.
+- Add regression coverage for the 65,536-token boundary and document the
+  dictionary size limit in the invariants and interchange format.
+
 ## [0.1.0](https://github.com/spiraldb/onpair/compare/v0.0.4...v0.1.0) - 2026-07-06
 
 ### Added

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "onpair"
-version = "0.1.0"
+version = "0.1.1"
 description = "Short-strings compression for fast random access"
 authors = ["SpiralDB Developers <hello@spiraldb.com>"]
 license = "Apache-2.0"

README.md

@@ -1,18 +1,84 @@
-# onpair
+# OnPair
 
-OnPair is a dictionary-based string compression algorithm designed for on-disk and in-memory database workloads that need both strong compression ratios and fast random access to individual values. 
-It builds its dictionary in a single sequential pass by incrementally merging frequent adjacent substrings, achieving compression comparable to BPE while being substantially faster and more memory-efficient. 
+[![Crates.io Version](https://img.shields.io/crates/v/onpair)](https://crates.io/crates/onpair)
+[![docs.rs](https://img.shields.io/docsrs/onpair)](https://docs.rs/onpair)
+[![CI](https://img.shields.io/github/actions/workflow/status/spiraldb/onpair/ci.yml?branch=develop)](https://github.com/spiraldb/onpair/actions/workflows/ci.yml)
+[![License](https://img.shields.io/crates/l/onpair)](LICENSE)
+[![MSRV](https://img.shields.io/crates/msrv/onpair)](https://github.com/spiraldb/onpair/blob/develop/rust-toolchain.toml)
 
-## Interchange format
+OnPair is a Rust codec for compressing string columns while keeping every
+row independently accessible. It populates a dictionary of up to 2^16 patterns
+recurring across the column, then encodes each row as a sequence of indices into
+it. Decoding just looks up each index and copies out the bytes, so columns
+decompress at high throughput, and any single row can be read or dropped without
+touching the ones around it.
 
-OnPair defines a shared in-memory representation — the *plain interchange form*
-that independent implementations exchange so a column produced by one is
-readable by another. It fixes the buffers (dictionary bytes, dictionary
-offsets, codes, and row offsets) and their invariants; denser internal
-encodings and on-disk serialization are out of scope. See
-[docs/interchange-format.md](docs/interchange-format.md).
+## Why OnPair
 
-## References
+- **Fast decompression.** The decoder looks up each index and copies out the
+  bytes, with no entropy-decoding stage in the hot path, so decoding sustains
+  high throughput.
+- **Random access.** Each row is a self-contained run of codes, so it decodes
+  on its own, with no block to unpack and no neighbours to rebuild.
+- **Efficient compression.** OnPair reaches ratios competitive with far heavier
+  methods while keeping its encoding pass fast, so strong compression comes at a
+  fraction of the usual cost.
+- **Compressed-domain search.** Evaluate equality, prefix, and substring
+  predicates without decoding rows back to bytes.
 
-- Paper: Francesco Gargiulo et al., *OnPair: Short Strings Compression for Fast Random Access* — [arXiv:2508.02280](https://arxiv.org/abs/2508.02280)
-- Reference C++ implementation: [gargiulofrancesco/onpair_cpp](https://github.com/gargiulofrancesco/onpair_cpp)
+OnPair is designed for high-cardinality columns, whose many distinct values tend
+to share substrings rather than repeat in full. On low-cardinality columns it is 
+better to deduplicate first: encode the column as references to the distinct 
+values, then run OnPair over that set of unique strings. Deduplication removes
+the value-level repetition, and OnPair compresses the substring redundancy that
+remains.
+
+## Benchmarks
+
+Benchmarks comparing OnPair against the main comparable string-compression
+algorithms are in progress.
+
+<!--
+TODO: Add numbers and plots comparing OnPair against the main similar codecs,
+across compression ratio, random-access latency, decompression throughput, and
+compressed-domain search.
+-->
+
+## Quick start
+
+```sh
+cargo add onpair
+```
+
+```rust
+use onpair::{compress, Config, DECODE_PADDING};
+
+// Three values in the Arrow layout OnPair takes: one byte buffer plus offsets.
+let values = b"catdogbird";
+let offsets = [0u32, 3, 6, 10]; // "cat", "dog", "bird"
+
+let column = compress(values, &offsets, Config::default()).unwrap();
+let view = column.view();
+
+// A value decodes to its original length, so a buffer of the longest value plus
+// DECODE_PADDING (which absorbs the decoder's 16-byte tail write) fits any row.
+let max_len = 4; // longest value in this column
+let mut decoded = Vec::<u8>::with_capacity(max_len + DECODE_PADDING);
+// SAFETY: capacity >= this row's decoded length + DECODE_PADDING.
+let len = unsafe { view.decompress_row_into(1, decoded.spare_capacity_mut()) };
+unsafe { decoded.set_len(len) }; // SAFETY: decode initialized `len` bytes.
+
+assert_eq!(decoded, b"dog");
+```
+
+`Config::default()` caps the dictionary at 2^12 entries; raise `max_dict_bits`
+for a larger one, up to 2^16.
+
+
+## Further reading
+
+- [In-memory interchange format](docs/interchange-format.md)
+- [OnPair: Short Strings Compression for Fast Random Access](https://arxiv.org/abs/2508.02280)
+- [C++ reference implementation](https://github.com/gargiulofrancesco/onpair_cpp)
+
+OnPair is licensed under [Apache-2.0](LICENSE).

benches/clickbench.rs

@@ -15,8 +15,7 @@
 //   1. env var `ONPAIR_BENCH_PARQUET` — path to a parquet file
 //      (e.g. ClickBench `hits.parquet`). Optionally set
 //      `ONPAIR_BENCH_COLUMN` to pick a specific UTF-8 column; otherwise
-//      we pick the first BYTE_ARRAY / Utf8 / Utf8View column with the
-//      largest total byte volume.
+//      we pick the first Utf8 / LargeUtf8 / Utf8View column in the schema.
 //   2. `/tmp/userdata1.parquet` if present (small real-world parquet,
 //      good for smoke runs).
 //   3. A synthetic ClickBench-shaped URL corpus (100 000 rows of