Skip to content

feat(stevec): typed query path — arrow returns ste_vec_entry, XOR-aware equality, stevec_query containment - #223

Merged
coderdan merged 12 commits into
mainfrom
feat/stevec-entry-typed-arrow
May 20, 2026
Merged

feat(stevec): typed query path — arrow returns ste_vec_entry, XOR-aware equality, stevec_query containment#223
coderdan merged 12 commits into
mainfrom
feat/stevec-entry-typed-arrow

Conversation

@coderdan

@coderdan coderdan commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes correctness gaps in the StEVec query path and adds the typed
infrastructure they need. Stacked on #221.

The single root cause behind several silent-wrong-result bugs was that
-> on eql_v2_encrypted returned a synthetic-root eql_v2_encrypted
rather than a typed sv entry. Cascading consequences:

  • WHERE col -> 'sel' = $1 resolved to the root-level = on the synthetic
    root, which extracts hm only — worked incidentally on hm-bearing
    selectors and silently returned zero rows on oc-bearing ones (string /
    number leaves).
  • WHERE col -> 'sel' < $1 resolved to the root-level < which extracts
    ob (Block ORE). sv entries have no ob, so the strict extractor raised.
  • ORDER BY eql_v2.ore_cllw(col -> 'sel') didn't compile because no
    (eql_v2_encrypted) overload of ore_cllw exists post-feat(eql_v2)!: collapse ste_vec ORE terms to single oc field #219. Callers
    needed a .data::eql_v2.ste_vec_entry cast workaround.

What lands

  • -> returns eql_v2.ste_vec_entry (all three overloads). Inlinable
    LANGUAGE sql bodies using jsonb_path_query_first / direct jsonb
    indexing. Preserves root i/v envelope metadata in the returned entry
    via the wider DOMAIN shape.
  • eql_v2.eq_term(ste_vec_entry) RETURNS bytea — XOR-aware equality
    term (coalesces hm and oc). New canonical extractor for field-level
    equality. = / <> on ste_vec_entry rewritten to reduce to
    eq_term(a) <op> eq_term(b). Range operators (<, <=, >, >=)
    unchanged — strictly via ore_cllw (XOR contract).
  • eql_v2.stevec_query DOMAIN — type-safe containment needle
    ({"sv": [...]} with no c field on any element). Companion cast
    to_stevec_query(eql_v2_encrypted). New operators:
    @>(eql_v2_encrypted, eql_v2.stevec_query),
    @>(eql_v2_encrypted, eql_v2.ste_vec_entry), plus <@ mirrors.
  • Removed: pre-release fused eql_v2.hmac_256(eql_v2_encrypted, text)
    — recipe migrates to eql_v2.eq_term(col -> '<sel>'). No internal
    callers; one dedicated test file deleted, replaced by eq_term_tests.rs.

Validation matrix (from docs/upgrading/v2.3.md — full notes there)

Recipe (post-PR) Engages
WHERE col -> 'sel' = $1::ste_vec_entry functional hash index on eq_term(col -> 'sel')
ORDER BY eql_v2.ore_cllw(col -> 'sel') LIMIT n functional btree (default opclass ore_cllw_ops)
WHERE col @> '{"sv":[...]}'::stevec_query GIN index on eql_v2.ste_vec(col)
WHERE col @> (other_col -> 'sel') same, via the entry-needle overload

Documentation

  • New upgrade notes U-007 (typed arrow selector) and U-008 (stevec_query).
  • U-004 recipe rewritten to the chained eq_term(col -> '<sel>') form.
  • Stale ORE example in U-005 fixed.
  • SteVecQueryPayload and SteVecQueryElement added to
    docs/reference/schema/eql-payload-v2.3.schema.json.
  • docs/reference/sql-support.md and database-indexes.md updated to
    the new recipe.
  • CHANGELOG.md Added / Changed / Removed entries.

Test plan

  • mise run build — main, Supabase, Protect variants all build.
  • mise run test — 39 test groups pass across PG 14 (locally).
    Includes 8 new eq_term_tests.rs tests, 5 new containment_with_index_tests
    for stevec_query, 5 new payload_schema_tests for SteVecQueryPayload,
    and ore_cllw_opclass_tests::functional_index_engages_via_arrow_chain
    (the load-bearing plan test for the ORDER BY recipe).
  • mise run docs:validate — 0 errors, 55 SQL files validated.
  • CI to confirm Postgres 14/15/16/17 + Supabase splinter green.

@coderabbitai

coderabbitai Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Rate limit exceeded

@coderdan has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 51 minutes and 28 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 24245f62-6cd4-4844-bc88-a239f3887117

📥 Commits

Reviewing files that changed from the base of the PR and between a04a1f2 and e5d6ce2.

📒 Files selected for processing (25)
  • CHANGELOG.md
  • docs/reference/database-indexes.md
  • docs/reference/schema/eql-payload-v2.3.schema.json
  • docs/reference/sql-support.md
  • docs/upgrading/v2.3.md
  • src/encrypted/hash.sql
  • src/jsonb/functions.sql
  • src/operators/->.sql
  • src/operators/<@.sql
  • src/operators/@>.sql
  • src/operators/ste_vec_entry.sql
  • src/ste_vec/eq_term.sql
  • src/ste_vec/types.sql
  • tasks/pin_search_path.sql
  • tasks/test/splinter.sh
  • tests/sqlx/src/assertions.rs
  • tests/sqlx/tests/comparison_tests.rs
  • tests/sqlx/tests/containment_tests.rs
  • tests/sqlx/tests/containment_with_index_tests.rs
  • tests/sqlx/tests/eq_term_tests.rs
  • tests/sqlx/tests/hmac_256_selector_tests.rs
  • tests/sqlx/tests/hmac_256_terms_tests.rs
  • tests/sqlx/tests/ore_cllw_opclass_tests.rs
  • tests/sqlx/tests/payload_schema_tests.rs
  • tests/sqlx/tests/specialized_tests.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/stevec-entry-typed-arrow

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderdan coderdan changed the title feat(stevec): typed query path — returns ste_vec_entry, XOR-aware equality, stevec_query containment feat(stevec): typed query path — arrow returns ste_vec_entry, XOR-aware equality, stevec_query containment May 19, 2026
@coderdan
coderdan force-pushed the feat/ore-cllw-opclass branch from 67b37f7 to 82ee9c7 Compare May 20, 2026 03:16
@coderdan
coderdan force-pushed the feat/stevec-entry-typed-arrow branch 3 times, most recently from 633be8c to 127f1ea Compare May 20, 2026 05:38
@coderdan
coderdan force-pushed the feat/ore-cllw-opclass branch from 2f8f76f to 8515527 Compare May 20, 2026 05:52
@coderdan
coderdan force-pushed the feat/stevec-entry-typed-arrow branch from 127f1ea to e1e9690 Compare May 20, 2026 05:52
@coderdan
coderdan force-pushed the feat/ore-cllw-opclass branch from 8515527 to ccbd49f Compare May 20, 2026 05:56
@coderdan
coderdan force-pushed the feat/stevec-entry-typed-arrow branch from e1e9690 to 28ec007 Compare May 20, 2026 05:56
@coderdan
coderdan force-pushed the feat/ore-cllw-opclass branch from ccbd49f to 1b0d08b Compare May 20, 2026 06:01
@coderdan
coderdan force-pushed the feat/stevec-entry-typed-arrow branch from 28ec007 to 9f38ecf Compare May 20, 2026 06:04
Base automatically changed from feat/ore-cllw-opclass to main May 20, 2026 06:06
coderdan added 11 commits May 20, 2026 16:07
XOR-aware equality term for `eql_v2.ste_vec_entry`. Returns the bytea
representation of whichever deterministic term the sv entry carries —
`hm` (HMAC-256, for bool leaves / array roots / object roots) or `oc`
(CLLW ORE, for string / number leaves). The two byte distributions
are disjoint by construction (different keys, different protocols),
so byte equality on the coalesce is unambiguous for a given selector.

This becomes the canonical equality extractor on ste_vec_entry —
follow-up commits rewrite `=` / `<>` to use it and switch index
recipes from `eql_v2.hmac_256(col -> 'sel')` to
`eql_v2.eq_term(col -> 'sel')`. Without this, `=` on an oc-bearing
selector (string / number leaves) silently returns zero rows because
`hmac_256(ste_vec_entry)` is NULL on both sides.
Under the XOR contract each sv entry carries exactly one of `hm` or
`oc` — bool leaves / array roots / object roots get `hm`, string /
number leaves get `oc`. The prior `=`/`<>` bodies reduced to
`hmac_256(a) = hmac_256(b)`, which silently NULLs (and therefore
returns zero rows) when applied to oc-bearing selectors.

Switch to `eq_term(a) = eq_term(b)` (and `<>` likewise). `eq_term`
coalesces hm/oc as bytea, so equality on either deterministic term
works structurally. Both byte distributions are disjoint by
construction, so the comparison is unambiguous.

The canonical functional-index recipe for field-level equality
shifts from `eql_v2.hmac_256(col -> 'sel')` to
`eql_v2.eq_term(col -> 'sel')` — one index covers both hm-bearing
and oc-bearing selectors. Documentation update follows in a later
commit.

Range operators (`<` / `<=` / `>` / `>=`) are unchanged: they keep
reducing to `ore_cllw(a) <op> ore_cllw(b)`. Range on hm-only entries
is meaningless and correctly produces silent NULL (lint subsystem
flags misconfigured indexes).
A `stevec_query` is the query-shaped payload used as the right-hand
needle for `@>` containment: a top-level `{"sv": [...]}` object
whose elements carry selector + index terms (`hm` or `oc`) but
never a ciphertext field (`c`). Typing the needle as `stevec_query`
documents at the API surface that containment matches against
indexes, not ciphertexts.

The DOMAIN CHECK enforces:
- Top-level object with an `sv` field that is a JSON array.
- No sv element may carry a `c` field (jsonb_path_exists rejects
  payloads with any `c`).

A companion cast `eql_v2.to_stevec_query(eql_v2_encrypted)` and the
`@>(eql_v2_encrypted, eql_v2.stevec_query)` overload land in
follow-up commits.
…edles

This is the centerpiece of the StEVec query-path correctness fix. The
single root cause behind several silent-wrong-result bugs was that `->`
returned the synthetic-root `eql_v2_encrypted` rather than a typed
`eql_v2.ste_vec_entry`. Cascading consequences:

- `WHERE col -> 'sel' = $1` resolved to root-level `=` on
  `eql_v2_encrypted`, which extracts `hm` from the synthetic root —
  worked incidentally on hm-bearing selectors and broke silently on
  oc-bearing ones (NULL = NULL → false).
- `WHERE col -> 'sel' < $1` resolved to root-level `<` which extracts
  `ob` (Block ORE). sv entries have no `ob`, so the strict extractor
  raised.
- `ORDER BY eql_v2.ore_cllw(col -> 'sel')` didn't compile because
  no `(eql_v2_encrypted)` overload of `ore_cllw` exists post-#219.
  Callers needed a `(col -> 'sel').data::ste_vec_entry` workaround.

This commit flips `->` (all three overloads — text selector, encrypted
selector, integer index) to `RETURNS eql_v2.ste_vec_entry`. The text
overload now uses `jsonb_path_query_first` over the sv array; the
integer overload uses direct jsonb indexing; both merge the root's
`{i, v}` envelope metadata into the returned entry. The DOMAIN CHECK
on `ste_vec_entry` tolerates extra fields, so the merged shape
`{i, v, s, c, hm-or-oc}` is valid, and per-entry extractors
(`eq_term`, `ore_cllw`, `selector`) ignore the extra `i`/`v`. All
three overloads are inlinable single-statement SQL.

With the typed selection in place, this commit also lands the typed
containment needles:

- `@>(eql_v2_encrypted, eql_v2.stevec_query)` — recommended recipe.
  `stevec_query` is the right-hand sv-shaped payload type (no `c`
  fields) defined in the previous commit. The operator delegates to
  `ste_vec_contains` after wrapping via `to_encrypted`.
- `@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` — convenience for
  "does this payload contain this specific entry?", e.g.
  `e @> (e -> 'sel')`. Wraps the entry into a single-element sv
  array (stripping `c`, which the containment logic ignores anyway).
- `<@` mirror overloads for both forms.

All new operators are allowlisted in `tasks/pin_search_path.sql` so
the post-install `ALTER FUNCTION ... SET search_path` pass doesn't
defeat inlinability.

Tests that depended on the prior synthetic-root return are updated:
the `term` text returned by helpers like `get_encrypted_term` is now
a JSON object literal (since `->` returns jsonb-domain), so it casts
naturally to `::eql_v2.ste_vec_entry`. The asymmetric-containment
test in `containment_tests.rs` is re-expressed via stevec_query (the
original `(entry) @> encrypted` shape is now type-prevented);
`specialized_tests.rs` similarly re-expresses through the typed
operators rather than direct `ste_vec_contains` calls.

`assertions.rs::QueryAssertion::count`/`returns_rows` now surface the
PG error string in the panic message — saves a round of bisection
when the SQL is shape-wrong rather than semantically wrong.

The synthetic-root caveat in the prior `->` doc comment is gone:
this commit *is* the refactor that comment foresaw.
…hain

The two-arg `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` was
a fused selector-match + hm-extract introduced as the migration path
off Blake3 for U-004. It bypasses the typed API and silently NULLs in
two distinct cases (selector miss, hm absent on the matched oc-bearing
element) without distinguishing them — which became actively
misleading once selectors can be either hm-bearing or oc-bearing under
the XOR contract.

Post the `->` flip and the new `eq_term` extractor, the canonical
recipe is `eql_v2.eq_term(col -> '<selector>')` — covers both hm and
oc selectors with one expression, and the chained form composes with
the rest of the typed model (operators, casts, containment).

Drops:
- `src/jsonb/functions.sql` — `hmac_256(eql_v2_encrypted, text)`
  function definition.
- `tasks/pin_search_path.sql` — allowlist entry for the dropped
  function.
- `tests/sqlx/tests/hmac_256_selector_tests.rs` — dedicated tests
  for the dropped function.

Adds equivalent coverage:
- `tests/sqlx/tests/eq_term_tests.rs` — happy path on both hm- and
  oc-bearing entries, STRICT NULL propagation, target-element pick
  in multi-entry sv arrays, functional hash index engagement for
  bare `WHERE` and `GROUP BY`.

Recipe migration in docs:
- `docs/reference/database-indexes.md` — per-selector hash index
  recipe updated to `eql_v2.eq_term(col -> '<sel>')`. Predicate
  shape simplifies to `WHERE col -> '<sel>' = $1::ste_vec_entry`.
- `src/encrypted/hash.sql` — `@note` rewritten to point at the new
  recipe (the post-#219 cast workaround is no longer needed).

Companion updates:
- `tests/sqlx/tests/hmac_256_terms_tests.rs` — `gin_containment_uses_index`
  now reads the entry's `hm` hex via JSONB field access on the new
  `ste_vec_entry`-typed `->` result, instead of the dropped fused form.
- `src/jsonb/functions.sql` — stale `@see hmac_256(eql_v2_encrypted, text)`
  on `hmac_256_terms` swapped for the active `eq_term` reference.

Remaining doc updates (v2.3.md upgrade notes, sql-support.md) follow
in a later commit alongside the broader changelog/upgrade work.
The `to_stevec_query(eql_v2_encrypted)` function was described in the
plan but missing from the earlier stevec_query commit. Adds it now:
takes an encrypted payload, strips `c` (ciphertext) from each sv
element, and returns the result as `eql_v2.stevec_query`. Registered
as an ASSIGNMENT cast so callers can write
`b.encrypted_doc::eql_v2.stevec_query` to convert another encrypted
column into a containment needle.

Adds the missing test coverage from the plan:

- `ore_cllw_opclass_tests::functional_index_engages_via_arrow_chain` —
  asserts `ORDER BY eql_v2.ore_cllw(col -> 'sel') LIMIT n` engages a
  functional btree index on the same expression (Index Scan, no Sort
  node). This is the load-bearing plan test for the recommended
  ordering recipe.

- `containment_with_index_tests` additions:
  - `stevec_query_domain_rejects_payloads_with_c`
  - `stevec_query_domain_rejects_non_sv_objects`
  - `stevec_query_domain_accepts_valid_payload`
  - `contains_with_stevec_query_overload` — end-to-end recipe test
  - `cast_eql_v2_encrypted_to_stevec_query_strips_c` — round-trip
    through the new cast function.
…cipe

- v2.3.md: rewrite U-004's recipe to the chained `eq_term(col -> 'sel')`
  form, drop the references to the removed fused hmac_256. Add U-007
  (typed `->` selector + `eq_term`) and U-008 (typed `stevec_query`
  containment). Fix the stale U-005 example that referenced the
  removed `eql_v2.ore_cllw(eql_v2_encrypted)` overload — the
  recommended form now uses `col -> '<sel>' < $1::ste_vec_entry`.
  Update U-006's strict-separation section to reflect that `->` now
  returns `ste_vec_entry` directly (no `.data::ste_vec_entry` cast).
- sql-support.md: bump the `->` row to reflect the new return type;
  update field-level GROUP BY recipe to `eq_term(col -> '<sel>')`.
- schema/eql-payload-v2.3.schema.json: add `SteVecQueryPayload` and
  `SteVecQueryElement` definitions for the containment-needle shape
  (no `c` field allowed on elements). Top-level oneOf is unchanged —
  query payloads aren't stored.
- payload_schema_tests: 5 new positive/negative tests for the new
  query-payload variant.
… removal

Updates the [Unreleased] section to reflect the typed StEVec query path:

- Added: `eql_v2.eq_term` extractor (XOR-aware), `eql_v2.stevec_query`
  DOMAIN + typed `@>` / `<@` overloads + `to_stevec_query` cast.
- Changed: `->` return type flipped to `eql_v2.ste_vec_entry` (all
  three overloads). Equality on `ste_vec_entry` now uses `eq_term`
  rather than `hmac_256`, picking up oc-bearing selectors that were
  silently returning zero rows.
- Removed: the pre-release fused
  `eql_v2.hmac_256(val eql_v2_encrypted, selector text)` — recipe
  migration to the chained `eq_term(col -> 'sel')` form.

Also rewrites the existing entries on `eql_v2.ste_vec_entry` and the
entry-type operators to reflect the new XOR-aware equality reduction
and to drop the "future PR may flip ->" caveat (this is that PR).
…ests

CI surfaced two regressions on the post-flip build:

- Supabase splinter (`function_search_path_mutable`): the new `->`
  overloads and `eq_term` are inlinable SQL (no SET clause, by
  design) so splinter correctly flags them. Add allowlist entries
  with the same rationale we used for the other typed extractors —
  pinning would break the functional-index match for the chained
  recipes.
- `mise run test:lint` (`cargo fmt --check`): the new
  containment_with_index_tests cases were not rustfmt-clean. Run
  cargo fmt to align.
Rewrites `@>(eql_v2_encrypted, eql_v2.stevec_query)` and
`@>(eql_v2_encrypted, eql_v2.ste_vec_entry)` to reduce to a native
`jsonb @>` over `eql_v2.to_stevec_query(a)::jsonb`. The planner now
matches the inlined predicate structurally against a functional GIN
index on the same expression, so the typed `@>` is the canonical
recipe end-to-end — no need to expose `to_stevec_query(col)` in the
WHERE clause to engage the index.

`to_stevec_query` tightens its element normalization to keep only
`{s, hm, oc}` (drops `c`, `a`, `i`, `v`, anything else cipherstash-suite
might emit). This is the matching-relevant set per the XOR contract;
both haystack and needle sides normalize identically so jsonb @>
compares apples-to-apples.

XOR-correctness gain: the previous `hmac_256_terms` recipe silently
dropped oc-bearing sv elements (string / number leaves carry `oc`,
not `hm`), so containment queries via that index could never match
on string / number selectors. The new recipe is XOR-aware — one
functional GIN covers every selector regardless of which term it
carries.

Adds explicit XOR-aware regression tests in
`containment_with_index_tests`:

- `typed_contains_matches_hm_bearing_selector` — sanity check
- `typed_contains_matches_oc_bearing_selector` — the load-bearing
  test for the XOR-correctness gap. Synthesises an sv with a single
  oc-bearing element and verifies `@>(stevec_query)` matches it.
  Would have silently returned zero rows under the
  hmac_256_terms-based recipe.
- `typed_contains_mixed_sv_engages_both_selector_kinds` — both
  hm and oc lookups against the same row
- `typed_contains_wrong_term_does_not_match` — negative case
- `functional_gin_on_to_stevec_query_engages_for_typed_contains` —
  the load-bearing plan assertion: GIN on
  `(eql_v2.to_stevec_query(col)::jsonb) jsonb_path_ops` is matched
  structurally by the inlined typed `@>(stevec_query)` body, so
  bare-form containment engages Bitmap Index Scan.

The needle is expected to be `{s, hm-or-oc}`-shaped per the
stevec_query contract; if callers pass through extracted entries
they should normalize first (the `contains_with_stevec_query_overload`
test demonstrates the pattern).
`eql_v2.hmac_256_terms(eql_v2_encrypted)` was added under PR #205
as the recommended GIN-indexable aggregate for field-level containment.
It's structurally wrong under the XOR contract: it filters out every
sv element lacking `hm`, which means every oc-bearing element (string /
number leaves) is invisible to the index. Containment queries via the
recipe could never match on string / number selectors.

The gap was masked by fixture data that violated the XOR contract
(several test selectors carried `hm` on string fields where
cipherstash-suite would emit `oc`). No test exercised hmac_256_terms
containment against a properly-emitted oc-bearing selector, so the
breakage was invisible.

Replacement: the typed `@>(eql_v2_encrypted, eql_v2.stevec_query)`
overload (added earlier in this PR, body inlined to a native jsonb
`@>` over `eql_v2.to_stevec_query(a)::jsonb` in the prior commit).
The canonical recipe is now:

  CREATE INDEX <name>
    ON <table> USING gin
       ((eql_v2.to_stevec_query(<col>)::jsonb) jsonb_path_ops);

  SELECT * FROM <table>
   WHERE <col> @> '{"sv":[{"s":"<sel>","hm-or-oc":"<term>"}]}'::eql_v2.stevec_query;

Coverage: `containment_with_index_tests::typed_contains_matches_oc_bearing_selector`
(added in the prior commit) is the regression-prevention test for the
gap — it synthesises a one-element sv with `oc` only and asserts the
typed @> matches it. Would silently return zero rows under the
hmac_256_terms recipe.

Drops:
- `src/jsonb/functions.sql` — function definition (lines 437-476)
  replaced by a comment explaining the removal and pointing at the
  replacement recipe.
- `tasks/pin_search_path.sql` — allowlist entry.
- `tasks/test/splinter.sh` — splinter allowlist row.
- `tests/sqlx/tests/hmac_256_terms_tests.rs` — dedicated test file
  (5 tests). Coverage of the actual containment recipe lives in
  `containment_with_index_tests::typed_contains_*` instead.
- `CHANGELOG.md` — `Added` entry for the function (pre-release, so
  net effect is "never existed in 2.3").

Doc updates:
- `docs/upgrading/v2.3.md` U-004 — recipe (b) rewritten to use
  `to_stevec_query`+ jsonb_path_ops GIN, query in typed `@>` form.
- `docs/reference/database-indexes.md` — same swap.
@coderdan
coderdan force-pushed the feat/stevec-entry-typed-arrow branch from 9f38ecf to 86c57ad Compare May 20, 2026 06:07
@freshtonic
freshtonic self-requested a review May 20, 2026 06:59

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very quick review but nothing is leaping out at me as an issue that would prevent merging. To keep @coderdan unblocked I will review in the background and issue a follow up with any changes if required.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes correctness gaps in the StEVec query path by making -> return a typed eql_v2.ste_vec_entry, introducing an XOR-aware equality extractor (eql_v2.eq_term), and adding a typed containment-needle DOMAIN (eql_v2.stevec_query) with corresponding @> / <@ overloads. It also removes the pre-release fused hmac_256(eql_v2_encrypted, text) path and updates tests/docs to the new typed recipes.

Changes:

  • Change -> (all overloads) to return eql_v2.ste_vec_entry with inlinable SQL bodies to preserve functional-index matching.
  • Add eql_v2.eq_term(ste_vec_entry) -> bytea and rewrite = / <> on ste_vec_entry to compare eq_term(...).
  • Add eql_v2.stevec_query + to_stevec_query cast and typed containment operators, with updated index recipes/tests/docs.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/sqlx/tests/specialized_tests.rs Updates containment tests to use typed -> and typed @> overloads.
tests/sqlx/tests/payload_schema_tests.rs Adds JSON-schema validation tests for SteVecQueryPayload/elements.
tests/sqlx/tests/ore_cllw_opclass_tests.rs Adds plan test ensuring functional index engagement via ore_cllw(value -> 'sel').
tests/sqlx/tests/hmac_256_terms_tests.rs Removes tests for deleted hmac_256_terms.
tests/sqlx/tests/hmac_256_selector_tests.rs Removes tests for deleted fused hmac_256(eql_v2_encrypted, text).
tests/sqlx/tests/eq_term_tests.rs Adds new tests for eq_term semantics and functional hash-index engagement.
tests/sqlx/tests/containment_with_index_tests.rs Adds stevec_query domain/operator tests and XOR-aware containment + index-engagement tests.
tests/sqlx/tests/containment_tests.rs Adjusts containment tests for new typed needle/entry shapes.
tests/sqlx/tests/comparison_tests.rs Updates comparisons to rely on typed ste_vec_entry results from ->.
tests/sqlx/src/assertions.rs Improves assertion failure output by including DB error details.
tasks/test/splinter.sh Updates splinter allowlist notes to reflect removed functions and new inlining-critical ones.
tasks/pin_search_path.sql Updates allowlist for inlining-critical functions/operators (->, eq_term, typed @>/<@) and removes deleted ones.
src/ste_vec/types.sql Adds stevec_query DOMAIN + to_stevec_query cast function.
src/ste_vec/eq_term.sql Introduces XOR-aware equality term extractor for ste_vec_entry.
src/operators/ste_vec_entry.sql Rewrites entry equality to use eq_term; updates operator docs accordingly.
src/operators/<@.sql Adds <@ overloads for stevec_query and ste_vec_entry LHS.
src/operators/@>.sql Adds @> overloads for stevec_query and ste_vec_entry RHS; implements typed containment via to_stevec_query(...)::jsonb @> ....
src/operators/->.sql Changes -> overloads to return ste_vec_entry and rewrites bodies to inlinable SQL.
src/jsonb/functions.sql Removes fused hmac_256(val, selector) and hmac_256_terms; adds rationale comment.
src/encrypted/hash.sql Updates guidance to use eq_term(col -> '<selector>') for field-level grouping.
docs/upgrading/v2.3.md Adds/updates upgrade notes for typed -> (U-007) and typed containment needle (U-008).
docs/reference/sql-support.md Updates supported recipes to use eq_term(col -> '<selector>') and notes -> now returns ste_vec_entry.
docs/reference/schema/eql-payload-v2.3.schema.json Adds SteVecQueryPayload and SteVecQueryElement definitions.
docs/reference/database-indexes.md Updates functional index recipes for equality and containment to eq_term/stevec_query.
CHANGELOG.md Updates Added/Changed/Removed entries to reflect eq_term, stevec_query, and removal of fused HMAC selector extractor.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/ste_vec/types.sql
Comment thread tests/sqlx/tests/containment_tests.rs Outdated
…nt test

Address Copilot review on #223:

1. The eql_v2.stevec_query DOMAIN CHECK only forbade `c` on sv elements.
   It now also requires each element to carry a selector `s` and exactly
   one deterministic term (`hm` XOR `oc`), matching the ste_vec_entry
   emission contract and the SteVecQueryElement JSON schema. Without
   this, a selector-only needle (`{"sv":[{"s":"x"}]}`) cast cleanly and
   then matched every row through the bare `jsonb @>` body.

2. contains_operator_term_does_not_contain_full_value asserted
   `e @> needle AND NOT (e @> e)` — `NOT (e @> e)` is always false, so
   the query returned 0 rows regardless of containment correctness.
   Rewrite it to assert genuine directional containment between two
   eql_v2_encrypted values: a one-entry subset of `e` is contained by
   `e`, but does not contain `e` back.

Add stevec_query DOMAIN reject tests for selector-only and
both-terms elements.
@coderdan
coderdan merged commit 7c858fd into main May 20, 2026
7 checks passed
@coderdan
coderdan deleted the feat/stevec-entry-typed-arrow branch May 20, 2026 07:50
tobyhede pushed a commit that referenced this pull request Jun 20, 2026
…nt test

Address Copilot review on #223:

1. The eql_v2.stevec_query DOMAIN CHECK only forbade `c` on sv elements.
   It now also requires each element to carry a selector `s` and exactly
   one deterministic term (`hm` XOR `oc`), matching the ste_vec_entry
   emission contract and the SteVecQueryElement JSON schema. Without
   this, a selector-only needle (`{"sv":[{"s":"x"}]}`) cast cleanly and
   then matched every row through the bare `jsonb @>` body.

2. contains_operator_term_does_not_contain_full_value asserted
   `e @> needle AND NOT (e @> e)` — `NOT (e @> e)` is always false, so
   the query returned 0 rows regardless of containment correctness.
   Rewrite it to assert genuine directional containment between two
   eql_v2_encrypted values: a one-entry subset of `e` is contained by
   `e`, but does not contain `e` back.

Add stevec_query DOMAIN reject tests for selector-only and
both-terms elements.
tobyhede pushed a commit that referenced this pull request Jun 20, 2026
feat(stevec): typed query path — arrow returns ste_vec_entry, XOR-aware equality, stevec_query containment
tobyhede pushed a commit that referenced this pull request Jun 20, 2026
The Upgrade notes paragraph said "Six numbered notes ... (U-006)", but
v2.3 has eight — U-007 (typed `->` selector lookup) and U-008 (typed
`stevec_query` containment), both added by #223, were already referenced
by the section's own Added entries. Also drop the mention of the removed
fused `eql_v2.hmac_256(col, '<selector>')` recipe from the U-004 summary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants