feat(eql_v2): btree operator class on eql_v2.ore_cllw - #221
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughAdds a btree operator family/class ChangesCLLW ORE btree operator class and index support
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bench validation ✅End-to-end validation on the bench Postgres (1M
Metadata sidecar: No Sort node; the btree walks in order; the LIMIT 10 closes after 10 rows. The other JSON scenarios ( |
3c061bf to
da6c2dc
Compare
da6c2dc to
67b37f7
Compare
67b37f7 to
82ee9c7
Compare
|
I noticed that there are errors when mise build generates the concatenated SQL files in the EQL repo. Looks like it’s still referencing the deleted files (on main anyway) Also, the build script should have failed but it seems to be swallowing errors. |
freshtonic
left a comment
There was a problem hiding this comment.
Approved, but conditional on whether the following things are not real concerns:
Concerns
1. compare_ore_cllw_term can return NULL — and it's now btree FUNCTION 1
src/ore_cllw/functions.sql:200-204 returns NULL when a.bytes IS NULL OR b.bytes IS NULL. This is reachable in practice: the (jsonb) overload of eql_v2.ore_cllw
(functions.sql:53-58) returns (bytes => NULL) when the input lacks oc, and the comment there says it does so deliberately for inlinability. PostgreSQL's btree FUNCTION 1
contract requires a non-NULL int for non-NULL composite inputs; NULL-at-the-composite-level is filtered by btree's null-handling layer, but non-NULL composite with NULL
field is not.
In a functional index built via eql_v2.ore_cllw((value).data), any row whose payload lacks oc would index a ROW(NULL)::eql_v2.ore_cllw value, and a subsequent
WHERE/ORDER BY traversal would call FUNCTION 1 with that as one operand. Behavior is undefined (likely silent misordering rather than a crash).
The test fixture seeds 20 rows all with oc present, so this path isn't exercised. Two suggestions, pick one:
- Add a fixture row with oc absent (or with bytes=NULL in the hand-crafted form) and assert either a graceful query result or an expected raise.
- Document the expectation in the opclass comment that callers must filter / partial-index out rows lacking oc, and recommend CREATE INDEX ... WHERE (value).data ? 'oc'
in docs/reference/database-indexes.md.
2. Tests cover ORDER BY but not WHERE <op> index match
The PR description and CHANGELOG both claim WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1::jsonb) engages Index Cond — but functional_index_engages_for_order_by
only verifies the ORDER BY ... LIMIT n shape. The WHERE form is the more common usage and a different planner path (range-quals against opclass strategies 1/2/4/5 vs.
strategy 3 equality). A parallel test asserting Index Cond on WHERE eql_v2.ore_cllw((value).data) < eql_v2.ore_cllw($1::jsonb) would close the gap and match the claims.
|
Build script fixed in e2fd2ad. Two compounding causes:
Fix:
Verified: clean build still passes; injecting a dangling Your two earlier review concerns (NULL handling in |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tasks/build.sh (1)
8-8: 💤 Low valueRemove the duplicate shebang.
Line 1 already declares the interpreter. This second shebang on line 8 has no effect and may confuse maintainers.
🧹 Proposed fix
-#!/bin/bash - set -euo pipefail🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tasks/build.sh` at line 8, Remove the duplicate shebang ("#!/bin/bash") found later in the script; keep the first interpreter declaration and delete the second occurrence to avoid confusion and redundancy in the tasks/build.sh script.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Line 27: The CHANGELOG entry currently ends with an issue link "(`#220`)";
update the end of that entry in CHANGELOG.md to reference the introducing PR
instead of the issue (replace the "(`#220`)" issue link with the PR link/number
for the PR that introduced this change) so the file follows the project's
convention of ending entries with the PR that introduced them.
In `@docs/upgrading/v2.3.md`:
- Line 176: The example uses the removed overload
eql_v2.ore_cllw(e->'<selector>'::text); update the doc to use the
ste_vec_entry-typed path instead so upgrade SQL remains valid: replace
references to eql_v2.ore_cllw and the removed eql_v2_encrypted overload with the
ste_vec_entry-typed expression (and keep the same extractor+operator semantics
and mention of eql_v2.ore_cllw_ops/default opclass) so the rewritten range query
shows the correct typed form for WHERE and ORDER BY ... LIMIT n.
---
Nitpick comments:
In `@tasks/build.sh`:
- Line 8: Remove the duplicate shebang ("#!/bin/bash") found later in the
script; keep the first interpreter declaration and delete the second occurrence
to avoid confusion and redundancy in the tasks/build.sh script.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9f9110d8-2bf1-42e7-bbae-c7f525fae74f
📒 Files selected for processing (9)
CHANGELOG.mddocs/reference/database-indexes.mddocs/upgrading/v2.3.mdsrc/ore_cllw/operator_class.sqlsrc/ore_cllw/operators.sqltasks/build.shtasks/pin_search_path.sqltasks/test/splinter.shtests/sqlx/tests/ore_cllw_opclass_tests.rs
| - **Columns configured with CLLW ORE (the consolidated `oc` field on sv elements)** — bare-form range queries on these columns will now raise. Two paths forward: | ||
| - **Preferred:** migrate the column configuration to `ore` (Block ORE) so the natural form works everywhere. | ||
| - **If you must keep the existing encoding:** rewrite range queries to the extractor form: `WHERE eql_v2.ore_cllw(e->'<selector>'::text) < eql_v2.ore_cllw($1::jsonb)`. The extractor is inlinable single-statement SQL, but functional-index match still requires an operator class on `eql_v2.ore_cllw` (tracked separately — see issue #220). | ||
| - **If you must keep the existing encoding:** rewrite range queries to the extractor form: `WHERE eql_v2.ore_cllw(e->'<selector>'::text) < eql_v2.ore_cllw($1::jsonb)`. The extractor + operators on `eql_v2.ore_cllw` are inlinable SQL, and the `eql_v2.ore_cllw_ops` btree opclass (DEFAULT FOR TYPE) lets a functional btree on `eql_v2.ore_cllw(e->'<selector>'::text)` engage for both `WHERE` and `ORDER BY ... LIMIT n` shapes. |
There was a problem hiding this comment.
The rewrite example uses a removed overload and should be updated to ste_vec_entry-typed form.
Line 176 documents eql_v2.ore_cllw(e->'<selector>'::text), but 2.3 migration guidance elsewhere says ore_cllw on eql_v2_encrypted was removed. This recipe should use the typed entry path to avoid broken upgrade SQL.
Proposed doc fix
- - **If you must keep the existing encoding:** rewrite range queries to the extractor form: `WHERE eql_v2.ore_cllw(e->'<selector>'::text) < eql_v2.ore_cllw($1::jsonb)`. The extractor + operators on `eql_v2.ore_cllw` are inlinable SQL, and the `eql_v2.ore_cllw_ops` btree opclass (DEFAULT FOR TYPE) lets a functional btree on `eql_v2.ore_cllw(e->'<selector>'::text)` engage for both `WHERE` and `ORDER BY ... LIMIT n` shapes.
+ - **If you must keep the existing encoding:** rewrite range queries to the typed-entry form: `WHERE eql_v2.ore_cllw((e->'<selector>'::text).data::eql_v2.ste_vec_entry) < eql_v2.ore_cllw(($1::jsonb)::eql_v2.ste_vec_entry)`. The extractor + operators on `eql_v2.ore_cllw` are inlinable SQL, and the `eql_v2.ore_cllw_ops` btree opclass (DEFAULT FOR TYPE) lets a functional btree on `eql_v2.ore_cllw((e->'<selector>'::text).data::eql_v2.ste_vec_entry)` engage for both `WHERE` and `ORDER BY ... LIMIT n` shapes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/upgrading/v2.3.md` at line 176, The example uses the removed overload
eql_v2.ore_cllw(e->'<selector>'::text); update the doc to use the
ste_vec_entry-typed path instead so upgrade SQL remains valid: replace
references to eql_v2.ore_cllw and the removed eql_v2_encrypted overload with the
ste_vec_entry-typed expression (and keep the same extractor+operator semantics
and mention of eql_v2.ore_cllw_ops/default opclass) so the rewritten range query
shows the correct typed form for WHERE and ORDER BY ... LIMIT n.
Addresses James's review concern on #221: btree FUNCTION 1 must return non-NULL int for non-NULL composite inputs. The previous extractor body returned `ROW(NULL)::eql_v2.ore_cllw` when the source payload lacked `oc`, which is a non-NULL composite with a NULL field — exactly the shape btree can't filter at the row-NULL layer. Indexed rows lacking `oc` would have landed `ROW(NULL)` into the index, and subsequent range queries would have called `compare_ore_cllw_term` with NULL bytes → undefined behaviour (silent misordering rather than a hard error). Two coupled changes: 1. `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and `eql_v2.ore_cllw(jsonb)` now return SQL-level NULL when the input lacks `oc`. Btree's standard null-handling then filters those rows: they never reach the comparator, range queries skip them by NULL semantics, ORDER BY sorts them at the NULLS LAST end. The extractor body becomes a CASE expression — still inlinable (PG inlines single-statement SQL functions with CASE). 2. `eql_v2.compare_ore_cllw_term` now RAISEs (not returns NULL) when a non-NULL composite has a NULL `bytes` field. Defense-in-depth: the extractor is designed to never emit that shape, so reaching this branch means a hand-crafted literal or a future regression. Loud failure beats silent misordering. The composite-level NULL guard stays as `RETURN NULL` for the non-index code paths. Documents the contract clearly in both function docstrings so the invariant is greppable. Closes James's review concern (1) on #221.
… match Adds coverage for the two concerns James raised on #221: 1. `ore_cllw_extractor_returns_null_when_oc_absent` / `ore_cllw_extractor_returns_composite_when_oc_present` — pin the new contract: extractor emits SQL-level NULL (not `ROW(NULL)`) when the input lacks `oc`. Covers both the `(jsonb)` and `(ste_vec_entry)` overloads. 2. `comparator_raises_on_null_bytes_in_non_null_composite` — pins the defense-in-depth behaviour: the comparator returns SQL NULL when fed a NULL composite (the normal path via the extractor) so range queries can compose correctly with btree's null-handling. The RAISE branch itself is currently unreachable under the single-field composite (composite-IS-NULL rule collapses ROW(NULL) to NULL), but the guard documents the invariant for future field additions. 3. `functional_index_engages_for_where_range` — closes James's concern (2). The existing `_for_order_by` test only covered ORDER BY; WHERE-clause range quals go through different planner paths (opclass strategies 1/2/4/5) and need separate coverage. Asserts both Index Scan / Bitmap Index Scan and Index Cond on the WHERE range. 4. `rows_without_oc_excluded_from_range_query` — end-to-end mixed-payload test: hm-only rows and oc-bearing rows in the same column. Verifies (a) no hm-only row appears in a range query result, and (b) the query completes without raising (which would happen if Option-A were bypassed and the extractor still emitted `ROW(NULL)`). Avoids asserting the exact count of oc-row matches because the CLLW protocol is adjacency-revealing rather than total-order-preserving — that semantics is outside this test's scope.
|
Both of your review concerns addressed in 175ae61 + 2f8f76f. Concern 1 —
|
… match Adds coverage for the two concerns James raised on #221: 1. `ore_cllw_extractor_returns_null_when_oc_absent` / `ore_cllw_extractor_returns_composite_when_oc_present` — pin the new contract: extractor emits SQL-level NULL (not `ROW(NULL)`) when the input lacks `oc`. Covers both the `(jsonb)` and `(ste_vec_entry)` overloads. 2. `comparator_raises_on_null_bytes_in_non_null_composite` — pins the defense-in-depth behaviour: the comparator returns SQL NULL when fed a NULL composite (the normal path via the extractor) so range queries can compose correctly with btree's null-handling. The RAISE branch itself is currently unreachable under the single-field composite (composite-IS-NULL rule collapses ROW(NULL) to NULL), but the guard documents the invariant for future field additions. 3. `functional_index_engages_for_where_range` — closes James's concern (2). The existing `_for_order_by` test only covered ORDER BY; WHERE-clause range quals go through different planner paths (opclass strategies 1/2/4/5) and need separate coverage. Asserts both Index Scan / Bitmap Index Scan and Index Cond on the WHERE range. 4. `rows_without_oc_excluded_from_range_query` — end-to-end mixed-payload test: hm-only rows and oc-bearing rows in the same column. Verifies (a) no hm-only row appears in a range query result, and (b) the query completes without raising (which would happen if Option-A were bypassed and the extractor still emitted `ROW(NULL)`). Avoids asserting the exact count of oc-row matches because the CLLW protocol is adjacency-revealing rather than total-order-preserving — that semantics is outside this test's scope.
2f8f76f to
8515527
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/sqlx/tests/ore_cllw_opclass_tests.rs (1)
346-373: 💤 Low valueConsider renaming test to reflect actual behavior.
The test name
comparator_raises_on_null_bytes_in_non_null_compositeis misleading—it actually verifies that a NULL composite flows through and returns NULL (not that it raises). The comment at lines 353-356 correctly notes that the RAISE branch is unreachable with the current single-field type definition.A clearer name would be
comparator_returns_null_for_null_compositeor similar, matching what the assertion actually checks.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/sqlx/tests/ore_cllw_opclass_tests.rs` around lines 346 - 373, The test function comparator_raises_on_null_bytes_in_non_null_composite is misnamed because it asserts that the comparator returns SQL NULL rather than raising; rename the test function to a clearer name such as comparator_returns_null_for_null_composite (update the async fn identifier) and change any references/comments that mention the old name so the test name matches the asserted behavior in the body and accompanying comment.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/sqlx/tests/ore_cllw_opclass_tests.rs`:
- Around line 346-373: The test function
comparator_raises_on_null_bytes_in_non_null_composite is misnamed because it
asserts that the comparator returns SQL NULL rather than raising; rename the
test function to a clearer name such as
comparator_returns_null_for_null_composite (update the async fn identifier) and
change any references/comments that mention the old name so the test name
matches the asserted behavior in the body and accompanying comment.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5b445c9b-4799-4525-8064-8427c8b2f419
📒 Files selected for processing (2)
src/ore_cllw/functions.sqltests/sqlx/tests/ore_cllw_opclass_tests.rs
Addresses James's review concern on #221: btree FUNCTION 1 must return non-NULL int for non-NULL composite inputs. The previous extractor body returned `ROW(NULL)::eql_v2.ore_cllw` when the source payload lacked `oc`, which is a non-NULL composite with a NULL field — exactly the shape btree can't filter at the row-NULL layer. Indexed rows lacking `oc` would have landed `ROW(NULL)` into the index, and subsequent range queries would have called `compare_ore_cllw_term` with NULL bytes → undefined behaviour (silent misordering rather than a hard error). Two coupled changes: 1. `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and `eql_v2.ore_cllw(jsonb)` now return SQL-level NULL when the input lacks `oc`. Btree's standard null-handling then filters those rows: they never reach the comparator, range queries skip them by NULL semantics, ORDER BY sorts them at the NULLS LAST end. The extractor body becomes a CASE expression — still inlinable (PG inlines single-statement SQL functions with CASE). 2. `eql_v2.compare_ore_cllw_term` now RAISEs (not returns NULL) when a non-NULL composite has a NULL `bytes` field. Defense-in-depth: the extractor is designed to never emit that shape, so reaching this branch means a hand-crafted literal or a future regression. Loud failure beats silent misordering. The composite-level NULL guard stays as `RETURN NULL` for the non-index code paths. Documents the contract clearly in both function docstrings so the invariant is greppable. Closes James's review concern (1) on #221.
… match Adds coverage for the two concerns James raised on #221: 1. `ore_cllw_extractor_returns_null_when_oc_absent` / `ore_cllw_extractor_returns_composite_when_oc_present` — pin the new contract: extractor emits SQL-level NULL (not `ROW(NULL)`) when the input lacks `oc`. Covers both the `(jsonb)` and `(ste_vec_entry)` overloads. 2. `comparator_raises_on_null_bytes_in_non_null_composite` — pins the defense-in-depth behaviour: the comparator returns SQL NULL when fed a NULL composite (the normal path via the extractor) so range queries can compose correctly with btree's null-handling. The RAISE branch itself is currently unreachable under the single-field composite (composite-IS-NULL rule collapses ROW(NULL) to NULL), but the guard documents the invariant for future field additions. 3. `functional_index_engages_for_where_range` — closes James's concern (2). The existing `_for_order_by` test only covered ORDER BY; WHERE-clause range quals go through different planner paths (opclass strategies 1/2/4/5) and need separate coverage. Asserts both Index Scan / Bitmap Index Scan and Index Cond on the WHERE range. 4. `rows_without_oc_excluded_from_range_query` — end-to-end mixed-payload test: hm-only rows and oc-bearing rows in the same column. Verifies (a) no hm-only row appears in a range query result, and (b) the query completes without raising (which would happen if Option-A were bypassed and the extractor still emitted `ROW(NULL)`). Avoids asserting the exact count of oc-row matches because the CLLW protocol is adjacency-revealing rather than total-order-preserving — that semantics is outside this test's scope.
8515527 to
ccbd49f
Compare
Restores functional-index match for sv-element ordered queries after the consolidation in #219 left the type without an opclass. Closes #220. `src/ore_cllw/operators.sql` — same-type comparison operators (`<`, `<=`, `=`, `>=`, `>`, `<>`) on `eql_v2.ore_cllw`. Each operator is backed by a single-statement `LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE` wrapper that reduces to `eql_v2.compare_ore_cllw_term(a, b) <op> 0`. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match. `src/ore_cllw/operator_class.sql` — `eql_v2.ore_cllw_ops` btree opclass registered `DEFAULT FOR TYPE eql_v2.ore_cllw`. FUNCTION 1 is `eql_v2.compare_ore_cllw_term` directly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase build variant via the existing `**/*operator_class.sql` glob in `tasks/build.sh` (operator classes require superuser). `tasks/pin_search_path.sql` — allowlists the six operator backing functions (`ore_cllw_eq` / `_neq` / `_lt` / `_lte` / `_gt` / `_gte`). Pinning would break the inlining chain and prevent the planner from structurally matching predicates to functional indexes. - **No `HASHES` / `MERGES` flags** on the operator declarations. HASHES needs a registered hash function on the type (no, and we don't want one — the CLLW protocol is for ordering, not hashing). MERGES needs an equivalent operator family on both sides, which we'd register separately if/when needed. This is the gap that disabled the pre-2025-06-24 opclass; see issue #220's history. - **Equality via `compare_ore_cllw_term = 0`**, not a `bytea_eq` shortcut. Consistent with the rest of the CLLW path; one source of truth for equality semantics; resilient to any future change in the underlying ciphertext encoding. - **The opclass operators are different from the operators on `eql_v2_encrypted`.** Those (per #211) inline to `ore_block_u64_8_256` and raise on non-Block-ORE columns. The new operators here are on the `eql_v2.ore_cllw` composite type itself — what callers reach through the extractor form `WHERE eql_v2.ore_cllw(col) <op> eql_v2.ore_cllw($1)`. No conflict, different scope. `tests/sqlx/tests/ore_cllw_opclass_tests.rs` covers: - Operator wiring: `=`, `<>`, `<`, `<=`, `>`, `>=` on hand-crafted byte strings under the CLLW per-byte protocol. - Cross-domain ordering via the leading tag byte (`0x00` numeric, `0x01` string) — numeric < string within the same column. - Opclass registration: `pg_opclass.opcdefault = true` for `eql_v2.ore_cllw_ops`. - Functional-index match: build a functional btree on `eql_v2.ore_cllw(value)`, confirm `EXPLAIN` for `ORDER BY eql_v2.ore_cllw(value) LIMIT n` shows `Index Scan` (or `Index Only Scan`) and no `Sort` node. - Inlinability lint: read `pg_proc` directly, assert each backing function is `LANGUAGE sql`, `IMMUTABLE`, `STRICT`, `PARALLEL SAFE`, and not pinned with `SET search_path`. From the bench results in cipherstash/benches#14 (post-#219 baseline): json/field_order/functional @ 1M = 20.0 s (no opclass; seq scan + Sort) With this opclass, `EXPLAIN` flips to `Index Scan + Limit` and the same query should land in single-digit ms on the bench rig. End-to-end bench re-run is the next step on the bench-side branch. Docs: CHANGELOG `Added` entry, U-005 action-required note refreshed, database-indexes.md ORE-CLLW recipe entry refreshed.
…class The new files introduced by the opclass — `src/ore_cllw/operators.sql` and `src/ore_cllw/operator_class.sql` — need `--! @file` / `--! @brief` file-level Doxygen comments to satisfy the 100% doc coverage gate that runs in `Test & Validate EQL (Postgres N)`. Convert the existing `--` block comments to `--!` Doxygen form so the existing content carries through into the generated XML; tighten the `@note` lines and add `@see` cross-references between the two files. Also adds Supabase splinter allowlist entries for the six inner comparators backing the new operators on `eql_v2.ore_cllw` — `ore_cllw_eq`, `ore_cllw_neq`, `ore_cllw_lt`, `ore_cllw_lte`, `ore_cllw_gt`, `ore_cllw_gte`. These are kept unpinned by `tasks/pin_search_path.sql` so they inline and the planner can carry the form through to the functional btree opclass match — same rationale as the existing `ore_block_u64_8_256_*` allowlist entries. Fixes the `Supabase splinter` and `Test & Validate EQL (Postgres 14-17)` jobs on this branch.
The file-level @brief satisfies docs:validate:coverage but the function-level required-tags lint also expects @brief + @return on each CREATE FUNCTION (within 50 lines of context). Add minimal Doxygen blocks per the existing ore_block_u64_8_256/operators.sql precedent. Fixes the Test & Validate EQL (Postgres 14-17) jobs on this branch.
Two fixes for `tests/sqlx/tests/ore_cllw_opclass_tests.rs` after the #219 strict-separation refactor merged into the base branch: - `functional_index_engages_for_order_by`: the index expression and ORDER BY clause referenced `eql_v2.ore_cllw(value)` (where `value` is an `eql_v2_encrypted` column). That overload was removed in #219; the typed replacement is `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` or the (jsonb) form. Switch both to `eql_v2.ore_cllw((value).data)`, which uses the (jsonb) overload and keeps the index/predicate expressions structurally identical so the planner still matches. - Remove the `ANALYZE ore_cllw_test` call. The `value` column is `eql_v2_encrypted` with payloads that carry only `oc` (no root `ob`). ANALYZE samples via the default btree opclass on `eql_v2_encrypted` (FUNCTION 1 = `eql_v2.compare`), and the post-#219 strict-Block-ORE `compare` raises on missing `ob`. The functional-index match still works without stats once `enable_seqscan = off` is set, so dropping ANALYZE is the cleanest path. Also refreshes `tests/sqlx/migrations/001_install_eql.sql` to the current built release SQL (this file is a build artefact that the mise test task copies from `release/`; keeping it in sync with the new opclass + entry types lets `cargo test` run against the up-to-date schema directly).
James spotted `cat: src/.../...sql: No such file or directory` lines during a local `mise run build`. Two compounding causes: 1. `src/deps.txt` / `src/deps-ordered.txt` were not in the cleanup block (only the supabase/protect variants were). Each successive build appended to them, so REQUIRE edges to long-deleted modules (blake3, ore_cllw_u64_8, ore_cllw_var_8, ope_cllw_u64_65, ope_cllw_var_8) sat there indefinitely. 2. `set -euxo pipefail` was commented out and `xargs cat`'s failures on missing files were swallowed, so the build still produced a release artefact + exited 0. This passed CI because fresh checkouts have no stale deps file — only locals (where the file accumulates across runs) ever saw it. Fix: - `set -euo pipefail` (dropped `x` to keep build output readable) - Add `rm -f` for `src/deps.txt`, `src/deps-ordered.txt`, and the appended `dbdev/eql--0.0.0.sql` - Add `verify_deps_exist` helper that scans the tsorted dep list and exits 1 with a pointed error message if any referenced file is missing. Runs after `tsort` for each variant. Verified locally: clean build still passes; injecting a dangling `-- REQUIRE: src/nonexistent/dummy.sql` makes the build exit 1 with `ERROR: src/deps-ordered.txt references missing file: ...`.
Addresses James's review concern on #221: btree FUNCTION 1 must return non-NULL int for non-NULL composite inputs. The previous extractor body returned `ROW(NULL)::eql_v2.ore_cllw` when the source payload lacked `oc`, which is a non-NULL composite with a NULL field — exactly the shape btree can't filter at the row-NULL layer. Indexed rows lacking `oc` would have landed `ROW(NULL)` into the index, and subsequent range queries would have called `compare_ore_cllw_term` with NULL bytes → undefined behaviour (silent misordering rather than a hard error). Two coupled changes: 1. `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and `eql_v2.ore_cllw(jsonb)` now return SQL-level NULL when the input lacks `oc`. Btree's standard null-handling then filters those rows: they never reach the comparator, range queries skip them by NULL semantics, ORDER BY sorts them at the NULLS LAST end. The extractor body becomes a CASE expression — still inlinable (PG inlines single-statement SQL functions with CASE). 2. `eql_v2.compare_ore_cllw_term` now RAISEs (not returns NULL) when a non-NULL composite has a NULL `bytes` field. Defense-in-depth: the extractor is designed to never emit that shape, so reaching this branch means a hand-crafted literal or a future regression. Loud failure beats silent misordering. The composite-level NULL guard stays as `RETURN NULL` for the non-index code paths. Documents the contract clearly in both function docstrings so the invariant is greppable. Closes James's review concern (1) on #221.
… match Adds coverage for the two concerns James raised on #221: 1. `ore_cllw_extractor_returns_null_when_oc_absent` / `ore_cllw_extractor_returns_composite_when_oc_present` — pin the new contract: extractor emits SQL-level NULL (not `ROW(NULL)`) when the input lacks `oc`. Covers both the `(jsonb)` and `(ste_vec_entry)` overloads. 2. `comparator_raises_on_null_bytes_in_non_null_composite` — pins the defense-in-depth behaviour: the comparator returns SQL NULL when fed a NULL composite (the normal path via the extractor) so range queries can compose correctly with btree's null-handling. The RAISE branch itself is currently unreachable under the single-field composite (composite-IS-NULL rule collapses ROW(NULL) to NULL), but the guard documents the invariant for future field additions. 3. `functional_index_engages_for_where_range` — closes James's concern (2). The existing `_for_order_by` test only covered ORDER BY; WHERE-clause range quals go through different planner paths (opclass strategies 1/2/4/5) and need separate coverage. Asserts both Index Scan / Bitmap Index Scan and Index Cond on the WHERE range. 4. `rows_without_oc_excluded_from_range_query` — end-to-end mixed-payload test: hm-only rows and oc-bearing rows in the same column. Verifies (a) no hm-only row appears in a range query result, and (b) the query completes without raising (which would happen if Option-A were bypassed and the extractor still emitted `ROW(NULL)`). Avoids asserting the exact count of oc-row matches because the CLLW protocol is adjacency-revealing rather than total-order-preserving — that semantics is outside this test's scope.
ccbd49f to
1b0d08b
Compare
Addresses James's review concern on #221: btree FUNCTION 1 must return non-NULL int for non-NULL composite inputs. The previous extractor body returned `ROW(NULL)::eql_v2.ore_cllw` when the source payload lacked `oc`, which is a non-NULL composite with a NULL field — exactly the shape btree can't filter at the row-NULL layer. Indexed rows lacking `oc` would have landed `ROW(NULL)` into the index, and subsequent range queries would have called `compare_ore_cllw_term` with NULL bytes → undefined behaviour (silent misordering rather than a hard error). Two coupled changes: 1. `eql_v2.ore_cllw(eql_v2.ste_vec_entry)` and `eql_v2.ore_cllw(jsonb)` now return SQL-level NULL when the input lacks `oc`. Btree's standard null-handling then filters those rows: they never reach the comparator, range queries skip them by NULL semantics, ORDER BY sorts them at the NULLS LAST end. The extractor body becomes a CASE expression — still inlinable (PG inlines single-statement SQL functions with CASE). 2. `eql_v2.compare_ore_cllw_term` now RAISEs (not returns NULL) when a non-NULL composite has a NULL `bytes` field. Defense-in-depth: the extractor is designed to never emit that shape, so reaching this branch means a hand-crafted literal or a future regression. Loud failure beats silent misordering. The composite-level NULL guard stays as `RETURN NULL` for the non-index code paths. Documents the contract clearly in both function docstrings so the invariant is greppable. Closes James's review concern (1) on #221.
… match Adds coverage for the two concerns James raised on #221: 1. `ore_cllw_extractor_returns_null_when_oc_absent` / `ore_cllw_extractor_returns_composite_when_oc_present` — pin the new contract: extractor emits SQL-level NULL (not `ROW(NULL)`) when the input lacks `oc`. Covers both the `(jsonb)` and `(ste_vec_entry)` overloads. 2. `comparator_raises_on_null_bytes_in_non_null_composite` — pins the defense-in-depth behaviour: the comparator returns SQL NULL when fed a NULL composite (the normal path via the extractor) so range queries can compose correctly with btree's null-handling. The RAISE branch itself is currently unreachable under the single-field composite (composite-IS-NULL rule collapses ROW(NULL) to NULL), but the guard documents the invariant for future field additions. 3. `functional_index_engages_for_where_range` — closes James's concern (2). The existing `_for_order_by` test only covered ORDER BY; WHERE-clause range quals go through different planner paths (opclass strategies 1/2/4/5) and need separate coverage. Asserts both Index Scan / Bitmap Index Scan and Index Cond on the WHERE range. 4. `rows_without_oc_excluded_from_range_query` — end-to-end mixed-payload test: hm-only rows and oc-bearing rows in the same column. Verifies (a) no hm-only row appears in a range query result, and (b) the query completes without raising (which would happen if Option-A were bypassed and the extractor still emitted `ROW(NULL)`). Avoids asserting the exact count of oc-row matches because the CLLW protocol is adjacency-revealing rather than total-order-preserving — that semantics is outside this test's scope.
feat(eql_v2): btree operator class on eql_v2.ore_cllw
Stacked on #219. Closes #220.
Summary
Restores the btree operator class on
eql_v2.ore_cllwso functional indexes engage for sv-element ordered queries. The pre-2025-06-24 opclass on the per-subtype CLLW types was disabled in69afdc8and never came back; the consolidation in #219 collapsed those types intoeql_v2.ore_cllwand this PR adds a clean opclass on the consolidated type.The gap was visible end-to-end in the bench data on cipherstash/benches#14:
json/field_order/functionalat 1M took 20 s (seq scan + Sort). With this opclass,EXPLAINflips toIndex Scan + Limit— expected sub-ms.What this adds
src/ore_cllw/operators.sql— same-type comparison operators (<,<=,=,>=,>,<>) oneql_v2.ore_cllw. Each backed by a single-statementLANGUAGE sql IMMUTABLE STRICT PARALLEL SAFEwrapper reducing toeql_v2.compare_ore_cllw_term(a, b) <op> 0. Wrappers inline so the planner can fold them into the calling query — that's what lets the index match.src/ore_cllw/operator_class.sql—eql_v2.ore_cllw_opsbtree opclass,DEFAULT FOR TYPE eql_v2.ore_cllw USING btree. FUNCTION 1 iseql_v2.compare_ore_cllw_termdirectly (plpgsql per-byte protocol; called only by btree internals, not per-row from the calling query). Excluded from the Supabase variant via the existing**/*operator_class.sqlglob intasks/build.sh.tasks/pin_search_path.sql— allowlists the six operator backing functions. Pinning would break the inlining chain.tests/sqlx/tests/ore_cllw_opclass_tests.rs— operator wiring, cross-domain ordering via the tag byte, opclass-is-default-for-type, functional-index match (EXPLAIN shows Index Scan, no Sort), inlinability lint.Design notes
HASHES/MERGESflags on the operator declarations. HASHES requires a registered hash function on the type (none, and we don't want one — CLLW is for ordering). MERGES requires a corresponding operator family registered on both sides. This was the gap #220's history identified as breaking the pre-2025-06-24 opclass.compare_ore_cllw_term = 0, not abytea_eqshortcut. One source of truth for equality semantics; resilient to any future change in ciphertext encoding.<,<=,>,>=oneql_v2_encrypted) inline toore_block_u64_8_256and raise on non-Block-ORE columns. The new operators here live on theeql_v2.ore_cllwcomposite type — what callers reach through the extractor form. No conflict, different scope.Test plan
mise run buildsucceeds for all three variants (main, supabase, protect).cipherstash-encrypt.sqlandcipherstash-encrypt-protect.sqlcontainCREATE OPERATOR CLASS eql_v2.ore_cllw_ops;cipherstash-encrypt-supabase.sqldoes not.mise run testgreen — 6 new opclass tests + 6 existing ORE-CLLW tests pass.pg_opclass.opcdefault = trueforeql_v2.ore_cllw_ops.eql_v2.ore_cllw(value)engagesIndex ScanforORDER BY ... LIMIT n(verified viaEXPLAINwithenable_seqscan=offon a 20-row fixture).field_order/*bench at 1M after merge — expected sub-ms via the new opclass.Merge order
This branch is stacked on #219. After #219 merges to main, rebase this onto main and flip to ready-for-review.
Summary by CodeRabbit
New Features
Documentation
Bug Fixes
Tests
Chores