From 8d267e8a54cf574fe9e4718b19d776aba340485e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Mon, 27 Jul 2026 19:41:27 +1000 Subject: [PATCH 1/5] docs: document eql_v3.grouped_value and fix the GROUP BY / DISTINCT shapes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EQL #423 added `eql_v3.grouped_value`, the aggregate that lets a query project an encrypted column while grouping by that column's equality term. Documenting it exposed two wrong claims on the grouping page: that `GROUP BY email` and `SELECT DISTINCT email` work in natural form, and that grouping compares equality terms. Neither is true — an encrypted column is a domain over jsonb with no custom operator class, so Postgres compares the raw (randomised) ciphertext and never collapses two encryptions of the same value. - Group on `eql_v3.eq_term(col)` for correctness first, planner economics second; drop the "raise work_mem if an ORM forces the raw form" advice, which rescued the performance of a query that was already returning wrong answers. - New `grouped_value` section: the 42803 rejection it fixes, the shape that fixes it, and its semantics (first non-null per group, all-NULL group is NULL, parallel safe, returns its input unchanged). - Deduplication is `DISTINCT ON (eql_v3.eq_term(col)) col` with the matching leading ORDER BY. Plain DISTINCT is not supported; `DISTINCT ON` has no projection restriction, so `grouped_value` is not involved there. - Same correction to the Indexes page's GROUP BY section, and grouped_value in the MIN/MAX and JSON-leaf examples. grouped_value landed after eql-3.0.2, so the page carries a version callout (with the 3.0.x join-back workaround) and the drift check gets an UNRELEASED allowlist — reported on every run, and deleted when EQL_RELEASE_TAG is bumped to a release whose manifest covers it. --- .../reference/eql/grouping-and-aggregates.mdx | 76 ++++++++++++++----- content/docs/reference/eql/index.mdx | 2 +- content/docs/reference/eql/indexes.mdx | 10 ++- scripts/generate-eql-api-docs.ts | 21 +++++ 4 files changed, 88 insertions(+), 21 deletions(-) diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index 8798d33..55c4785 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -1,6 +1,6 @@ --- title: Grouping & aggregates -description: "GROUP BY, DISTINCT, COUNT, and eql_v3.min/max on encrypted columns — why to group on the extractor, and why SUM and AVG stay client-side." +description: "GROUP BY on the equality term, eql_v3.grouped_value, DISTINCT ON, COUNT, and eql_v3.min/max on encrypted columns — and why SUM and AVG stay client-side." type: reference components: [eql] verifiedAgainst: @@ -9,32 +9,71 @@ verifiedAgainst: -Grouping and deduplication need an equality term, so they work on the same variants as `=`: `_eq`, every `_ord` variant, and the `text_search` variants. `MIN` / `MAX` need an ordering term (any `_ord` variant, or `text_search`). Arithmetic aggregates don't work at all — that's the last section. As everywhere, operands and call-site casts must be typed; see [Core concepts](/reference/eql/core-concepts). +Grouping and deduplication key on the **equality term**, never on the stored ciphertext, so they work on the same variants as `=`: `_eq`, every `_ord` variant, and the `text_search` variants. `MIN` / `MAX` need an ordering term (any `_ord` variant, or `text_search`). Arithmetic aggregates don't work at all — that's the last section. As everywhere, operands and call-site casts must be typed; see [Core concepts](/reference/eql/core-concepts). -## `GROUP BY` and `DISTINCT` +## Group on the equality term -Both work in natural form on equality-capable variants: +Encryption is non-deterministic: the same plaintext encrypts to a different ciphertext every time. An encrypted column is a domain over `jsonb`, and Postgres groups a domain using the *base type's* comparison — so `GROUP BY email` compares whole encrypted payloads and puts almost every row in a group of its own. There's no error, just wrong answers. + +Group on the equality term instead. It's a deterministic keyed hash, identical for equal plaintexts: ```sql -SELECT email, COUNT(*) FROM logins GROUP BY email; -SELECT DISTINCT email FROM logins; +SELECT COUNT(*) + FROM logins + GROUP BY eql_v3.eq_term(email); ``` -Grouping compares equality terms, so rows encrypting the same plaintext land in the same group — but the group key that comes back is ciphertext. Decrypt it in the client if you need to display it. +The term is an HMAC — opaque and one-way — so there's rarely a reason to select it. To get the encrypted value back alongside the aggregate, use [`eql_v3.grouped_value`](#grouped-value). -## Group on the extractor +Grouping on the extractor is also what makes the query fast. `GROUP BY email` would use the entire encrypted payload — 1–2 KB per row — as the hash key: Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is small, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. The same reasoning, from the index-tuning angle, is in [Indexes](/reference/eql/indexes). -For anything beyond small tables, group on the equality-term extractor instead of the raw column: +## Projecting the encrypted value: `eql_v3.grouped_value` [#grouped-value] + +Once you group by the term, Postgres rejects a bare reference to the column — it can't prove the column is constant within each group: ```sql -SELECT eql_v3.eq_term(email) AS email_term, COUNT(*) +SELECT email, COUNT(*) -- ERROR: column "logins.email" must appear in the + FROM logins -- GROUP BY clause or be used in an aggregate + GROUP BY eql_v3.eq_term(email); -- function (SQLSTATE 42803) +``` + +Every row in a group encrypts the same plaintext, so any one of them represents the group. `eql_v3.grouped_value` is the aggregate that hands one back: + +```sql +SELECT eql_v3.grouped_value(email) AS email, COUNT(*) FROM logins GROUP BY eql_v3.eq_term(email); ``` -The reason is planner economics. `GROUP BY email` uses the entire encrypted payload — 1–2 KB per row — as the hash key. Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is a small deterministic term: the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. If an ORM forces the raw-column form, raising `work_mem` is the rescue knob — but the extractor form is the design. The same reasoning, from the index-tuning angle, is in [Indexes](/reference/eql/indexes). +What comes back is a real encrypted payload, which the client decrypts as usual — unlike the group key itself, which is a term and can never be decrypted. + +```sql +eql_v3.grouped_value(jsonb) RETURNS jsonb +``` + +One aggregate covers every encrypted type: all the domains are jsonb-backed, and `grouped_value` returns its input unchanged — it never decrypts, inspects, or compares the value. Three details worth knowing: + +- It returns the **first non-`NULL`** value in each group. Which value is "first" is arbitrary without an intra-aggregate `ORDER BY`, and not deterministic under parallel aggregation — that's fine here, because every member of the group is an encryption of the same plaintext. +- A group of all-`NULL` values aggregates to `NULL`, matching native aggregate semantics. +- It's `PARALLEL SAFE` with a combine function, so it survives partial aggregation on large `GROUP BY` workloads. + + +**Requires an EQL release later than 3.0.2.** `eql_v3.grouped_value` was added in [encrypt-query-language#423](https://github.com/cipherstash/encrypt-query-language/pull/423) and ships in the next EQL release; it re-creates the `grouped_value` aggregate the EQL v2 surface had. On 3.0.x, group by the term and join the grouped result back to the table on that term to recover a representative encrypted value, then decrypt that in the client. + + +## Deduplicating: `DISTINCT ON` the equality term + +Plain `SELECT DISTINCT email` is not supported, for the same reason: it deduplicates raw ciphertext, so two encryptions of one value are never collapsed and you get every row back. + +Deduplicate on the term with `DISTINCT ON`. Unlike `GROUP BY`, it puts no restriction on the projection, so you select the encrypted column directly and no aggregate is involved: + +```sql +SELECT DISTINCT ON (eql_v3.eq_term(email)) email + FROM logins + ORDER BY eql_v3.eq_term(email); +``` -Note the trade-off: grouping on `eq_term` returns the *term*, not the encrypted value — fine for counting, but the term itself can't be decrypted. If you need the group key's plaintext, join the grouped result back to the table on the term to recover a representative encrypted value, then decrypt that in the client. +The leading `ORDER BY` expression has to match the `DISTINCT ON` expression — Postgres requires it — and the row that survives each group is the first one in that order, so add further sort keys after it to choose which. That's the no-aggregate path; reach for `grouped_value` when you also want a per-group aggregate like `COUNT(*)`. ## `COUNT` and `COUNT(DISTINCT)` @@ -44,7 +83,7 @@ Plain `COUNT(col)` counts non-`NULL` rows — it never compares values, so it wo SELECT COUNT(tax_id) FROM users; -- works even on bare public.eql_v3_text ``` -`COUNT(DISTINCT col)` deduplicates, so it needs an equality-capable variant — and the same extractor advice applies: +`COUNT(DISTINCT col)` deduplicates, so it has the ciphertext problem too: on the raw column it counts distinct ciphertexts, which is just the row count. Count distinct terms instead: ```sql SELECT COUNT(DISTINCT eql_v3.eq_term(email)) FROM logins; @@ -69,8 +108,10 @@ Comparison routes through the variant's `<` / `>` operator on the ordering term SELECT eql_v3.min(salary) FROM users; SELECT eql_v3.max(salary) FROM users WHERE department = 'engineering'; --- Combined with grouping -SELECT eql_v3.eq_term(department_code) AS dept, eql_v3.max(salary) +-- Combined with grouping: grouped_value returns the department code encrypted, +-- so the client can decrypt it to label the row. +SELECT eql_v3.grouped_value(department_code) AS department_code, + eql_v3.max(salary) FROM users GROUP BY eql_v3.eq_term(department_code); ``` @@ -94,12 +135,13 @@ A btree on `eql_v3.ord_term(col)` serves `MIN` / `MAX` — the [Indexes](/refere Leaves inside an encrypted JSON document group the same way — extract the entry by selector, then group on its equality term: ```sql -SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) +SELECT eql_v3.grouped_value(metadata -> 'region_selector'::text) AS region, + COUNT(*) FROM orders GROUP BY eql_v3.eq_term(metadata -> 'region_selector'::text); ``` -`eql_v3.eq_term` reads whichever term the entry carries, so this works on every JSON node type. String and Number leaves also support `eql_v3.min` / `eql_v3.max` via their CLLW OPE term. Selectors and node capabilities are in [JSON](/reference/eql/json). +An extracted entry is a `public.eql_v3_jsonb_entry`, jsonb-backed like every other encrypted domain, so `grouped_value` and `DISTINCT ON (eql_v3.eq_term(...))` both apply unchanged. `eql_v3.eq_term` reads whichever term the entry carries, so this works on every JSON node type. String and Number leaves also support `eql_v3.min` / `eql_v3.max` via their CLLW OPE term. Selectors and node capabilities are in [JSON](/reference/eql/json). ## Where to go next diff --git a/content/docs/reference/eql/index.mdx b/content/docs/reference/eql/index.mdx index 693645a..92291db 100644 --- a/content/docs/reference/eql/index.mdx +++ b/content/docs/reference/eql/index.mdx @@ -173,7 +173,7 @@ EQL v3 is designed to install without superuser. There are no custom operator cl `ORDER BY` on encrypted columns, and how to keep the sort in the index. - `GROUP BY`, `DISTINCT`, `COUNT`, and the `MIN` / `MAX` aggregates. + `GROUP BY` on the equality term, `DISTINCT ON`, `COUNT`, and the `MIN` / `MAX` aggregates. Equijoins on encrypted columns and the same-keyset rule. diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index d518cbd..1074747 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -115,15 +115,19 @@ If you `SELECT col::jsonb ... ORDER BY col`, Postgres folds the cast into the sc ### GROUP BY and DISTINCT -Group on the extractor, not the raw column: +Group and deduplicate on the extractor, not the raw column: ```sql -SELECT eql_v3.eq_term(email), count(*) +SELECT eql_v3.grouped_value(email) AS email, count(*) FROM users GROUP BY eql_v3.eq_term(email); + +SELECT DISTINCT ON (eql_v3.eq_term(email)) email + FROM users + ORDER BY eql_v3.eq_term(email); ``` -`GROUP BY email` uses the entire encrypted payload (1–2 KB per row) as the hash key; Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is a small deterministic term, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. If an ORM forces the raw-column form, raising `work_mem` is the rescue knob — but the extractor form is the design. +The extractor form is a correctness requirement before it's a tuning one: encryption is non-deterministic and the column is a domain over `jsonb`, so `GROUP BY email` and `SELECT DISTINCT email` compare raw ciphertext and never collapse two encryptions of the same value. It's also the faster shape — `GROUP BY email` uses the entire encrypted payload (1–2 KB per row) as the hash key, so Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`, while the small deterministic term fits in `work_mem` and gets a `HashAggregate`. Full treatment in [Grouping and aggregates](/reference/eql/grouping-and-aggregates). ## Encrypted JSON diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index 609954d..c895923 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -443,6 +443,7 @@ function driftCheck(manifest: Manifest): string[] { const unknown: string[] = []; for (const [fqn, pages] of referenced) { if (known.has(fqn) || MANIFEST_BLIND_SPOTS.has(fqn)) continue; + if (UNRELEASED.has(fqn)) continue; unknown.push(`${fqn} (in ${[...pages].join(", ")})`); } return unknown.sort(); @@ -461,6 +462,22 @@ function driftCheck(manifest: Manifest): string[] { // them. const MANIFEST_BLIND_SPOTS = new Set(["eql_v3.query_text_eq"]); +// Symbols merged into EQL but not yet in the pinned release, documented ahead of +// it because a page would otherwise teach a workaround for a solved problem. The +// manifest can't resolve them, so they're allowlisted here — and reported on +// every run, so the list stays visible rather than becoming a quiet exemption. +// +// The page documenting one MUST carry a version callout saying which release it +// needs. Delete the entry when EQL_RELEASE_TAG (scripts/generate-eql-docs.ts) is +// bumped to a release that ships it: the manifest covers it from then on, and +// the drift check goes back to being the authority. +const UNRELEASED = new Map([ + [ + "eql_v3.grouped_value", + "added after eql-3.0.2 in cipherstash/encrypt-query-language#423; documented in grouping-and-aggregates.mdx", + ], +]); + // ── Main ───────────────────────────────────────────────────────────────────── function main() { const manifest = loadManifest(); @@ -482,6 +499,10 @@ function main() { `✓ Generated ${path.relative(process.cwd(), OUT_FILE)} from EQL ${manifest.version} (${manifest.functions.length} functions)`, ); + for (const [fqn, why] of UNRELEASED) { + console.log(`• Drift check: ${fqn} allowlisted as unreleased — ${why}`); + } + const unknown = driftCheck(manifest); if (unknown.length) { const header = `⚠ ${unknown.length} schema-qualified symbol(s) referenced in hand-written pages are not in the manifest (domains or functions):`; From 2376fe973304dcbf5dfd71383a40d41395fc6f8b Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Mon, 27 Jul 2026 22:15:15 +1000 Subject: [PATCH 2/5] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20rest?= =?UTF-8?q?ore=20heading,=20flag=20broken=20examples,=20drop=20version=20n?= =?UTF-8?q?ote?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Restore the `GROUP BY` and `DISTINCT` heading, and fold deduplication back under it: one section for how both operators must be written, with the `grouped_value` projection following as its own section. - Add a `BadExample` component (rustdoc `compile_fail` styling: red frame, explicit label, verbatim error) so a broken example can't be mistaken for a working one while skimming, and use it for the two broken shapes here — the natural `GROUP BY` / `DISTINCT` forms, and the bare column under GROUP BY. - Drop the unreleased-version callout. --- .../reference/eql/grouping-and-aggregates.mdx | 60 ++++++++++--------- src/components/bad-example.tsx | 48 +++++++++++++++ src/mdx-components.tsx | 2 + 3 files changed, 83 insertions(+), 27 deletions(-) create mode 100644 src/components/bad-example.tsx diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index 55c4785..8fa7ef2 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -11,32 +11,54 @@ verifiedAgainst: Grouping and deduplication key on the **equality term**, never on the stored ciphertext, so they work on the same variants as `=`: `_eq`, every `_ord` variant, and the `text_search` variants. `MIN` / `MAX` need an ordering term (any `_ord` variant, or `text_search`). Arithmetic aggregates don't work at all — that's the last section. As everywhere, operands and call-site casts must be typed; see [Core concepts](/reference/eql/core-concepts). -## Group on the equality term +## `GROUP BY` and `DISTINCT` -Encryption is non-deterministic: the same plaintext encrypts to a different ciphertext every time. An encrypted column is a domain over `jsonb`, and Postgres groups a domain using the *base type's* comparison — so `GROUP BY email` compares whole encrypted payloads and puts almost every row in a group of its own. There's no error, just wrong answers. +Encryption is non-deterministic: the same plaintext encrypts to a different ciphertext every time. An encrypted column is a domain over `jsonb`, and Postgres compares a domain using the *base type's* comparison — so the natural forms compare whole encrypted payloads, never the plaintext behind them. They raise no error; they just answer wrongly. -Group on the equality term instead. It's a deterministic keyed hash, identical for equal plaintexts: + + +```sql +SELECT email, COUNT(*) FROM logins GROUP BY email; -- one group per row +SELECT DISTINCT email FROM logins; -- every row comes back +``` + + + +Key on the equality term instead. It's a deterministic keyed hash, identical for equal plaintexts: ```sql SELECT COUNT(*) FROM logins GROUP BY eql_v3.eq_term(email); + +SELECT DISTINCT ON (eql_v3.eq_term(email)) email + FROM logins + ORDER BY eql_v3.eq_term(email); ``` -The term is an HMAC — opaque and one-way — so there's rarely a reason to select it. To get the encrypted value back alongside the aggregate, use [`eql_v3.grouped_value`](#grouped-value). +Plain `DISTINCT` has nowhere to put the term, so deduplication is `DISTINCT ON`. Postgres requires the leading `ORDER BY` expression to match the `DISTINCT ON` expression, and the row that survives each group is the first one in that order — add further sort keys after it to choose which. + +The term is an HMAC — opaque and one-way — so there's rarely a reason to select it. `DISTINCT ON` puts no restriction on the projection, so the encrypted column comes back alongside it for the client to decrypt; under `GROUP BY` that takes [`eql_v3.grouped_value`](#grouped-value). -Grouping on the extractor is also what makes the query fast. `GROUP BY email` would use the entire encrypted payload — 1–2 KB per row — as the hash key: Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is small, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. The same reasoning, from the index-tuning angle, is in [Indexes](/reference/eql/indexes). +Keying on the extractor is also what makes the query fast. `GROUP BY email` would use the entire encrypted payload — 1–2 KB per row — as the hash key: Postgres estimates a hash table far larger than the default `work_mem` and falls back to a disk-spilling `GroupAggregate`. The extractor key is small, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably. The same reasoning, from the index-tuning angle, is in [Indexes](/reference/eql/indexes). -## Projecting the encrypted value: `eql_v3.grouped_value` [#grouped-value] +## `eql_v3.grouped_value` [#grouped-value] -Once you group by the term, Postgres rejects a bare reference to the column — it can't prove the column is constant within each group: +Grouping by the term costs you the column: Postgres rejects a bare reference to it, because it can't prove the column is constant within each group. + + ```sql -SELECT email, COUNT(*) -- ERROR: column "logins.email" must appear in the - FROM logins -- GROUP BY clause or be used in an aggregate - GROUP BY eql_v3.eq_term(email); -- function (SQLSTATE 42803) +SELECT email, COUNT(*) + FROM logins + GROUP BY eql_v3.eq_term(email); ``` + + Every row in a group encrypts the same plaintext, so any one of them represents the group. `eql_v3.grouped_value` is the aggregate that hands one back: ```sql @@ -57,23 +79,7 @@ One aggregate covers every encrypted type: all the domains are jsonb-backed, and - A group of all-`NULL` values aggregates to `NULL`, matching native aggregate semantics. - It's `PARALLEL SAFE` with a combine function, so it survives partial aggregation on large `GROUP BY` workloads. - -**Requires an EQL release later than 3.0.2.** `eql_v3.grouped_value` was added in [encrypt-query-language#423](https://github.com/cipherstash/encrypt-query-language/pull/423) and ships in the next EQL release; it re-creates the `grouped_value` aggregate the EQL v2 surface had. On 3.0.x, group by the term and join the grouped result back to the table on that term to recover a representative encrypted value, then decrypt that in the client. - - -## Deduplicating: `DISTINCT ON` the equality term - -Plain `SELECT DISTINCT email` is not supported, for the same reason: it deduplicates raw ciphertext, so two encryptions of one value are never collapsed and you get every row back. - -Deduplicate on the term with `DISTINCT ON`. Unlike `GROUP BY`, it puts no restriction on the projection, so you select the encrypted column directly and no aggregate is involved: - -```sql -SELECT DISTINCT ON (eql_v3.eq_term(email)) email - FROM logins - ORDER BY eql_v3.eq_term(email); -``` - -The leading `ORDER BY` expression has to match the `DISTINCT ON` expression — Postgres requires it — and the row that survives each group is the first one in that order, so add further sort keys after it to choose which. That's the no-aggregate path; reach for `grouped_value` when you also want a per-group aggregate like `COUNT(*)`. +Deduplicating without a per-group aggregate needs none of this — that's `DISTINCT ON`, above. ## `COUNT` and `COUNT(DISTINCT)` diff --git a/src/components/bad-example.tsx b/src/components/bad-example.tsx new file mode 100644 index 0000000..4da2ca2 --- /dev/null +++ b/src/components/bad-example.tsx @@ -0,0 +1,48 @@ +import { CircleX } from "lucide-react"; + +interface BadExampleProps { + /** Short label for the header strip. */ + label?: string; + /** Verbatim error the example produces, rendered as a footer. */ + error?: string; + /** The example itself (a fenced code block). */ + children: React.ReactNode; +} + +/** + * A code example that does NOT work, styled so a reader skimming the page can + * never mistake it for one that does — the failure mode of showing broken SQL + * in the same grey box as the fix. + * + * Modelled on rustdoc's `compile_fail` blocks: red frame, an explicit label, + * and (optionally) the verbatim error underneath. The example keeps the site's + * syntax highlighting and copy button, because a reader reproducing the failure + * is a legitimate thing to want. + */ +export function BadExample({ + label = "Doesn't work", + error, + children, +}: BadExampleProps) { + return ( +
+
+
+ +
{children}
+ + {error ? ( +

+ {error} +

+ ) : null} +
+ ); +} diff --git a/src/mdx-components.tsx b/src/mdx-components.tsx index 17e7aa1..57411a1 100644 --- a/src/mdx-components.tsx +++ b/src/mdx-components.tsx @@ -2,6 +2,7 @@ import { Callout } from "fumadocs-ui/components/callout"; import { Step, Steps } from "fumadocs-ui/components/steps"; import defaultMdxComponents from "fumadocs-ui/mdx"; import type { MDXComponents } from "mdx/types"; +import { BadExample } from "@/components/bad-example"; import { TrackedCodeBlock } from "@/components/code-block"; import { EqlFn } from "@/components/eql-fn"; import { EqlVersion } from "@/components/eql-version"; @@ -16,6 +17,7 @@ export function getMDXComponents(components?: MDXComponents): MDXComponents { Callout, Steps, Step, + BadExample, EqlVersion, EqlFn, ZeroKmsRegions, From cab3fc2d0be99639a187fb346815fe1019cbf8ca Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Mon, 27 Jul 2026 22:42:11 +1000 Subject: [PATCH 3/5] docs: pin EQL 3.0.3 and fix the fuzzy-match and encrypted-JSON surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documenting U-011 (the empty-bloom fuzzy-match guard, shipped in 3.0.3) meant moving EQL_RELEASE_TAG off 3.0.0, and the strict drift check then reported what the pin had been hiding: 3.0.1 renamed the whole encrypted-JSON surface and replaced the text match operator, and the hand-written pages still taught the 3.0.0 spelling of both. Every customer on a released 3.0.x has been reading docs whose text-match and JSON examples raise. Text fuzzy match (U-008, U-011): - `@>` / `<@` and `eql_v3.contains` / `contained_by` on text_match, text_search and text_search_ore are now `@@` / `eql_v3.matches`; the old spellings raise, and the pages say so. - Document the empty-needle guard as a LIKE truth table: a sub-floor search term (no trigrams, `bf: []`) used to match every row and now matches only empty-bloom values. - Note the one regression it carries: `matches` is non-STRICT and names the needle twice, so a needle from an uncorrelated subquery stops inlining and loses the GIN index. Bind parameters and literals are unaffected. Encrypted JSON (U-007, U-010): - eql_v3_json → eql_v3_json_search, eql_v3_jsonb_entry → eql_v3_json_entry, query_jsonb → query_json; the bare eql_v3_json is now storage-only. - Field equality is value-selector containment, not `=` on an extracted leaf (which raises). `eq_term` on an entry is a deprecated alias for `ope_term` and is not an exact equality term — so JSON leaves can no longer be grouped by equality at all, which the grouping page now states outright. - Wire format: document-level key header `h`, no root `c`, selector-derived nonces, sentinel value entries, `hm` retired. - `jsonb_array_elements_text` was removed; entry ciphertext is not independently decryptable. Tooling: - The fragment generator keyed the match card off a `match` capability the 3.0.1 catalog no longer emits (text_match now reports `["storage"]` with `supportedOperators: ["@@"]`), so the text function catalog silently lost its match entry. It now keys off the operator and renders `eql_v3.matches`. - Drift check: exempt lines that name a symbol in order to say it is gone, so a rename note can name the old symbol — the same line-local exemption validate-content-api.ts makes. - Two more manifest blind spots, both verified present in the 3.0.3 install SQL: `eql_v3.grouped_value` (hand-written CREATE AGGREGATE; the catalog only contributes the 60 generated min/max aggregates) and `public.eql_v3_json` (created in a DO block, like the query-operand domains). Generated artifacts (functions.mdx, the partials, eql-version.ts, the stack EQL API page) are left alone: prebuild regenerates them from the pinned release, and the committed copies are placeholders — the stack one still says EQL 2.3.0-pre.3. --- content/docs/reference/eql/core-concepts.mdx | 30 ++-- content/docs/reference/eql/filtering.mdx | 27 ++-- .../reference/eql/grouping-and-aggregates.mdx | 19 ++- content/docs/reference/eql/indexes.mdx | 23 ++- content/docs/reference/eql/json.mdx | 145 ++++++++++-------- content/docs/reference/eql/text.mdx | 50 ++++-- scripts/generate-eql-api-docs.ts | 115 ++++++++------ scripts/generate-eql-docs.ts | 10 +- 8 files changed, 246 insertions(+), 173 deletions(-) diff --git a/content/docs/reference/eql/core-concepts.mdx b/content/docs/reference/eql/core-concepts.mdx index 2b6ada8..fb6a239 100644 --- a/content/docs/reference/eql/core-concepts.mdx +++ b/content/docs/reference/eql/core-concepts.mdx @@ -4,7 +4,7 @@ description: "The model behind every EQL page: domain variants that declare capa type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- @@ -26,11 +26,11 @@ For any scalar type ``, the family looks like this: | `public.eql_v3__ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | | `public.eql_v3__ord_ope` | The byte-identical twin of `_ord`, with OPE pinned. See [SEM specifiers](#sem-specifiers). | | `public.eql_v3__ord_ore` | As `_ord`, with block-ORE pinned. | -| `public.eql_v3_text_match` (text only) | Free-text token containment: `@>` / `<@`. | -| `public.eql_v3_text_search` (text only) | Equality + ordering + token containment. | +| `public.eql_v3_text_match` (text only) | Free-text fuzzy match: `@@`. | +| `public.eql_v3_text_search` (text only) | Equality + ordering + fuzzy match. | | `public.eql_v3_text_search_ore` (text only) | As `text_search`, with block-ORE pinned. | -Every public domain name carries the `eql_v3_` prefix. It keeps EQL's types from shadowing built-in Postgres type names such as `text` and `json`, and gives each EQL version its own column-type namespace so two versions can coexist. Query-operand domains live in the versioned `eql_v3` schema already, so they are unprefixed: `eql_v3.query_text_eq`, `eql_v3.query_jsonb`. +Every public domain name carries the `eql_v3_` prefix. It keeps EQL's types from shadowing built-in Postgres type names such as `text` and `json`, and gives each EQL version its own column-type namespace so two versions can coexist. Query-operand domains live in the versioned `eql_v3` schema already, so they are unprefixed: `eql_v3.query_text_eq`, `eql_v3.query_json`. Two things worth calling out: @@ -70,7 +70,7 @@ CREATE TABLE users ( ); ``` -Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `public.eql_v3_json`, with its own operator surface — see [JSON](/reference/eql/json). +Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.3 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use separate domains — `public.eql_v3_json_search` for searchable documents, `public.eql_v3_json` for storage-only ones — with their own operator surface; see [JSON](/reference/eql/json). ## The three schemas @@ -117,9 +117,9 @@ Alongside the envelope, a payload carries the index terms for its column's capab | Key | SEM type | Wire shape | Enables | Reveals | | --- | --- | --- | --- | --- | | `hm` | `eql_v3_internal.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | -| `op` | `eql_v3_internal.ope_cllw` (domain over `bytea`) | Hex-encoded CLLW OPE ciphertext | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ope` domains and `text_search`, and on String / Number leaves of `public.eql_v3_json` | The relative order of two values | +| `op` | `eql_v3_internal.ope_cllw` (domain over `bytea`) | Hex-encoded CLLW OPE ciphertext | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ope` domains and `text_search`, and on String / Number leaves of `public.eql_v3_json_search` | The relative order of two values | | `ob` | `eql_v3_internal.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | The same comparisons on the pinned `_ord_ore` and `text_search_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | -| `bf` | `eql_v3_internal.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | +| `bf` | `eql_v3_internal.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@@` fuzzy match on `_match` and `text_search` domains | Probabilistic token overlap between values | The capability is encoded as **required keys**: the payload for a `public.eql_v3_text_eq` column must carry `hm`; a `public.eql_v3_integer_ord` payload must carry `op` (and only `op`); a `text_match` payload must carry `bf`; a `text_search` payload carries `hm`, `op`, and `bf`. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. @@ -139,9 +139,9 @@ A scalar payload for a `public.eql_v3_text_search` column (lookup + ordering + f - `v`, `i`, `c` — the envelope - `hm` — equality term: `WHERE email = $1` compares this - `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks -- `bf` — bloom-filter term: `@>` token containment tests these bit positions +- `bf` — bloom-filter term: `@@` fuzzy match tests these bit positions -Encrypted JSON documents use a different payload shape — an `sv` array with one encrypted entry per path in the document instead of a root ciphertext — defined in [JSON](/reference/eql/json). +Encrypted JSON documents use a different payload shape — a document-level key header `h` and an `sv` array with one encrypted entry per path, instead of a root ciphertext — defined in [JSON](/reference/eql/json). ### Machine-readable schemas @@ -154,12 +154,16 @@ The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publ The `eql_v3` domains are backed by `jsonb`. When an operand has no known type — a bare string literal, an untyped parameter — PostgreSQL reduces the domain to its `jsonb` base type and resolves the **native `jsonb` operator** instead of the encrypted one. The query doesn't fail; it silently returns native `jsonb` semantics, which are meaningless for encrypted payloads. + + ```sql --- ❌ Wrong: untyped parameter. PostgreSQL falls back to the native jsonb `=`, --- which compares raw payloads — syntactically valid, semantically meaningless. SELECT * FROM users WHERE email = $1; +``` --- ✅ Right: typed operand — the encrypted `=` resolves. + + +```sql +-- Typed operand — the encrypted `=` resolves. SELECT * FROM users WHERE email = $1::public.eql_v3_text_eq; ``` @@ -179,7 +183,7 @@ SELECT * FROM users WHERE salary > $1::public.eql_v3_bigint_eq; A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. (A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` themselves always work, on every variant.) -`LIKE` and `ILIKE` are blocked on **every** encrypted variant — pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment instead; [Text](/reference/eql/text) covers it. +`LIKE` and `ILIKE` are blocked on **every** encrypted variant — pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter fuzzy match (`@@`) instead; [Text](/reference/eql/text) covers it. One equality subtlety follows from the term table above, and it splits on whether the column is text. diff --git a/content/docs/reference/eql/filtering.mdx b/content/docs/reference/eql/filtering.mdx index f81a6bf..b7f282b 100644 --- a/content/docs/reference/eql/filtering.mdx +++ b/content/docs/reference/eql/filtering.mdx @@ -4,7 +4,7 @@ description: "WHERE-clause patterns on encrypted columns: equality, IN lists, ra type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- @@ -60,33 +60,32 @@ WHERE occurred_at >= $1::public.eql_v3_timestamp_ord AND occurred_at < $2::public.eql_v3_timestamp_ord; ``` -## Text token matching: `@>` +## Text fuzzy matching: `@@` -There is no `LIKE` on encrypted columns — encrypted free-text matching is bloom-filter token containment via `@>` on a `text_match` or `text_search` column: +There is no `LIKE` on encrypted columns — encrypted free-text matching is bloom-filter fuzzy match via `@@` on a `text_match` or `text_search` column: ```sql -SELECT * FROM users WHERE name @> $1::public.eql_v3_text_match; +SELECT * FROM users WHERE name @@ $1::public.eql_v3_text_match; ``` -The client encrypts the search term into a bloom-filter query value; matching is probabilistic (false positives possible, false negatives not). For the full no-`LIKE` story and match-term tuning, see [Text](/reference/eql/text). +The client encrypts the search term into a bloom-filter query value; matching is probabilistic (false positives possible, false negatives not). `@>` / `<@` raise on these domains — they were the old spelling of this match and are now blockers. For the full no-`LIKE` story, short-term behaviour, and match-term tuning, see [Text](/reference/eql/text). ## JSON containment and path filters -Encrypted JSON documents (`public.eql_v3_json`) filter by containment and path existence: +Encrypted JSON documents (`public.eql_v3_json_search`) filter by containment and path existence: ```sql -- Does the document contain this (encrypted) structure? -SELECT * FROM orders WHERE metadata @> $1::eql_v3.query_jsonb; +SELECT * FROM orders WHERE metadata @> $1::eql_v3.query_json; -- Does this path exist in the document? SELECT * FROM orders WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector'); --- Equality on an extracted leaf -SELECT * FROM orders -WHERE metadata -> 'email_selector'::text = $1::public.eql_v3_jsonb_entry; +-- Exact match on a field: containment on a value selector, built by the client +SELECT * FROM orders WHERE metadata @> $1::eql_v3.query_json; ``` -Field access is by selector hash, not plaintext path. The full JSON surface — containment, field access, path queries, and range filters on extracted leaves — is in [JSON](/reference/eql/json). +Field access is by selector hash, not plaintext path — and exact field matching is containment on a **value selector**, not `=` on an extracted leaf, which raises. The full JSON surface — containment, field access, path queries, and range filters on extracted leaves — is in [JSON](/reference/eql/json). ## Combining predicates @@ -97,7 +96,7 @@ SELECT * FROM users WHERE status = 'active' -- plaintext column, native operator AND created_at >= $1::public.eql_v3_timestamp_ord -- encrypted range AND (email = $2::public.eql_v3_text_eq -- encrypted equality - OR name @> $3::public.eql_v3_text_match); -- encrypted token match + OR name @@ $3::public.eql_v3_text_match); -- encrypted fuzzy match ``` The planner treats each encrypted predicate independently, so it can combine an index on a plaintext column with a functional index on an encrypted one (bitmap-AND, or whichever plan is cheapest). @@ -119,8 +118,8 @@ Don't confuse this with a JSON `null` *inside* an encrypted document, which is a | --- | --- | --- | --- | | Equality | `=` `<>` `IN` | `_eq`, all `_ord` variants, `text_search` variants | hash (or btree) on `eql_v3.eq_term` — btree on `eql_v3.ord_term` for the non-text `_ord` variants | | Range | `<` `<=` `>` `>=` `BETWEEN` | all `_ord` variants, `text_search` variants | btree on `eql_v3.ord_term` (`eql_v3.ord_term_ore` on `_ord_ore`) | -| Text token match | `@>` `<@` | `text_match`, `text_search` variants | GIN on `eql_v3.match_term` | -| JSON containment | `@>` `<@` | `public.eql_v3_json` | GIN on `eql_v3.to_ste_vec_query(col)::jsonb` | +| Text fuzzy match | `@@` | `text_match`, `text_search` variants | GIN on `eql_v3.match_term` | +| JSON containment | `@>` `<@` | `public.eql_v3_json_search` | GIN on `eql_v3.to_ste_vec_query(col)::jsonb` | | Null check | `IS NULL` / `IS NOT NULL` | every variant | — | Every one of these has a full index recipe — which method, which extractor, and how to confirm the index engages with `EXPLAIN` — in [Indexes](/reference/eql/indexes). diff --git a/content/docs/reference/eql/grouping-and-aggregates.mdx b/content/docs/reference/eql/grouping-and-aggregates.mdx index 8fa7ef2..809a689 100644 --- a/content/docs/reference/eql/grouping-and-aggregates.mdx +++ b/content/docs/reference/eql/grouping-and-aggregates.mdx @@ -4,7 +4,7 @@ description: "GROUP BY on the equality term, eql_v3.grouped_value, DISTINCT ON, type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- @@ -15,7 +15,7 @@ Grouping and deduplication key on the **equality term**, never on the stored cip Encryption is non-deterministic: the same plaintext encrypts to a different ciphertext every time. An encrypted column is a domain over `jsonb`, and Postgres compares a domain using the *base type's* comparison — so the natural forms compare whole encrypted payloads, never the plaintext behind them. They raise no error; they just answer wrongly. - + ```sql SELECT email, COUNT(*) FROM logins GROUP BY email; -- one group per row @@ -47,7 +47,7 @@ Keying on the extractor is also what makes the query fast. `GROUP BY email` woul Grouping by the term costs you the column: Postgres rejects a bare reference to it, because it can't prove the column is constant within each group. @@ -136,18 +136,17 @@ A btree on `eql_v3.ord_term(col)` serves `MIN` / `MAX` — the [Indexes](/refere **`SUM`, `AVG`, and every other arithmetic aggregate are unsupported** on encrypted columns — they would require homomorphic encryption, which EQL does not do. `MIN` / `MAX` work because they only need *comparison*, which the ordering term provides. For sums and averages, select the rows (or `MIN`/`MAX`/`COUNT` server-side to narrow them) and aggregate client-side after decryption. -## Grouping on extracted JSON leaves +## Leaves inside encrypted JSON -Leaves inside an encrypted JSON document group the same way — extract the entry by selector, then group on its equality term: +Leaves extracted from an encrypted JSON document (`metadata -> 'region_selector'::text`, a `public.eql_v3_json_entry`) **cannot be grouped by equality**. Entries stopped carrying an equality term in EQL 3.0.1: exact matching on a JSON field became presence of a value selector, which containment answers as a filter — there is no per-field key to hash rows into groups with. + +What still works on an extracted leaf is ordering: `eql_v3.min` / `eql_v3.max` on String and Number leaves, via their CLLW OPE term. ```sql -SELECT eql_v3.grouped_value(metadata -> 'region_selector'::text) AS region, - COUNT(*) - FROM orders - GROUP BY eql_v3.eq_term(metadata -> 'region_selector'::text); +SELECT eql_v3.min(metadata -> 'total_selector'::text) FROM orders; ``` -An extracted entry is a `public.eql_v3_jsonb_entry`, jsonb-backed like every other encrypted domain, so `grouped_value` and `DISTINCT ON (eql_v3.eq_term(...))` both apply unchanged. `eql_v3.eq_term` reads whichever term the entry carries, so this works on every JSON node type. String and Number leaves also support `eql_v3.min` / `eql_v3.max` via their CLLW OPE term. Selectors and node capabilities are in [JSON](/reference/eql/json). +To group by a field, promote it to its own encrypted column — a `text_eq` (or any `_ord`) column alongside the document — and group on that column's equality term as above. Selectors, value selectors, and node capabilities are in [JSON](/reference/eql/json). ## Where to go next diff --git a/content/docs/reference/eql/indexes.mdx b/content/docs/reference/eql/indexes.mdx index 1074747..6313b49 100644 --- a/content/docs/reference/eql/indexes.mdx +++ b/content/docs/reference/eql/indexes.mdx @@ -4,7 +4,7 @@ description: "Create Postgres indexes on encrypted columns using functional inde type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- @@ -16,7 +16,7 @@ EQL indexes are ordinary PostgreSQL functional indexes over **term-extractor fun | `eql_v3.eq_term(col)` | `hash` (or `btree`) | `hm` (HMAC-256) | equality | | `eql_v3.ord_term(col)` | `btree` | `op` (CLLW OPE) | range, `ORDER BY`, `MIN` / `MAX` | | `eql_v3.ord_term_ore(col)` | `btree` | `ob` (ORE block) | the same, on pinned `_ord_ore` / `text_search_ore` columns | -| `eql_v3.match_term(col)` | `gin` | `bf` (bloom filter) | text containment | +| `eql_v3.match_term(col)` | `gin` | `bf` (bloom filter) | text fuzzy match (`@@`) | The extractors are inlinable SQL functions, so the planner rewrites a bare-form predicate into the same expression the index was built on. You don't rewrite queries to use the index: @@ -51,7 +51,7 @@ CREATE INDEX users_created_at_ord CREATE INDEX users_created_at_ord_ore ON users USING btree (eql_v3.ord_term_ore(created_at_ore)); --- Text match: GIN index on match_term +-- Text fuzzy match (`@@`): GIN index on match_term -- (columns typed public.eql_v3_text_match or text_search) CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(name)); @@ -67,7 +67,7 @@ Create indexes when the table has a significant number of rows (typically more t All three must hold: -1. **The value carries the required term.** Range needs `op` (or `ob` on the pinned block-ORE variants), containment needs `bf`, and equality needs `hm`, except on the non-text `_ord` variants, where equality resolves against the ordering term instead. Which terms travel in a value's payload is decided by the encryption client — a value with only a bloom term will not drive an equality index. +1. **The value carries the required term.** Range needs `op` (or `ob` on the pinned block-ORE variants), fuzzy match needs `bf`, and equality needs `hm`, except on the non-text `_ord` variants, where equality resolves against the ordering term instead. Which terms travel in a value's payload is decided by the encryption client — a value with only a bloom term will not drive an equality index. 2. **The index was built after the data carried the term.** If you change which terms a column's values carry, recreate the index. 3. **The query operand is typed.** A typed parameter (`$1`, which CipherStash Proxy supplies) or an explicit cast resolves the encrypted operator; a bare `jsonb` literal falls through to native `jsonb` semantics and skips the index entirely: @@ -90,6 +90,19 @@ SELECT * FROM users WHERE email = $1::public.eql_v3_text_eq; -- Index Cond: (eql_v3.eq_term(email) = eql_v3.eq_term($1)) ``` +### Fuzzy match + +`col @@ $1` inlines to bloom array containment over `eql_v3.match_term(col)`, so the GIN index engages: + +```sql +SELECT * FROM users WHERE name @@ $1::public.eql_v3_text_match; +-- Bitmap Index Scan on users_name_match +``` + + +**Supply the needle as a bind parameter or a literal.** Since EQL 3.0.3 the match carries an empty-needle guard (see [Text](/reference/eql/text)), which makes `eql_v3.matches` non-`STRICT` and references the needle twice. A needle supplied as an *uncorrelated subquery* — `WHERE name @@ (SELECT …)` — therefore no longer inlines, and the query falls back to a sequential scan. The result is still correct; only the index acceleration is lost. + + ### Range and ORDER BY The `<`, `<=`, `>`, `>=` operators inline to comparisons on `eql_v3.ord_term`, so natural-form range predicates match the btree: @@ -131,7 +144,7 @@ The extractor form is a correctness requirement before it's a tuning one: encryp ## Encrypted JSON -Containment (`@>` / `<@`) on `public.eql_v3_json` document columns uses a GIN index over `eql_v3.to_ste_vec_query(col)::jsonb`, and field-level equality and ordering have their own extractor recipes. See [JSON](/reference/eql/json). +Containment (`@>` / `<@`) on `public.eql_v3_json_search` document columns uses a GIN index over `eql_v3.to_ste_vec_query(col)::jsonb` — the same index that serves exact field matching, which is containment on a value selector. Field-level ordering has its own extractor recipe. See [JSON](/reference/eql/json). ## Verify with EXPLAIN diff --git a/content/docs/reference/eql/json.mdx b/content/docs/reference/eql/json.mdx index e2aea00..0db66b0 100644 --- a/content/docs/reference/eql/json.mdx +++ b/content/docs/reference/eql/json.mdx @@ -1,61 +1,78 @@ --- title: JSON -description: "The complete reference for encrypted JSON documents with public.eql_v3_json — the ste_vec payload shape, containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." +description: "The complete reference for encrypted JSON documents with public.eql_v3_json_search — the ste_vec payload shape, value-selector containment, field access, and path queries over ciphertext, with the native jsonb operators that don't apply blocked outright." type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- -`public.eql_v3_json` is EQL's encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, and equality or range comparisons on extracted leaves. +`public.eql_v3_json_search` is EQL's searchable encrypted JSON document type, built on structured encryption (**ste_vec**). The document is encrypted as a vector of encrypted entries — one entry per path inside the document — and every path is queryable without decryption: containment, field and array access, exact field matching, and range comparisons on extracted leaves. -Like every EQL type, `public.eql_v3_json` holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. +Like every EQL type, it holds ciphertext the database can't read. Encryption, decryption, and selector generation happen in the client — the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy). See [Searchable encryption](/concepts/searchable-encryption) for how querying ciphertext works at all. ## The types -Three `jsonb`-backed domains make up the encrypted JSON surface: +Four `jsonb`-backed domains make up the encrypted JSON surface. The family follows the same convention as the scalars: the bare name is storage-only, and a suffix adds capability. | Type | What it is | | --- | --- | -| `public.eql_v3_json` | The column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | -| `public.eql_v3_jsonb_entry` | A single entry from the vector: a selector, a ciphertext, and exactly one index term. This is what `->` returns. | -| `eql_v3.query_jsonb` | A containment needle: entries with selectors and index terms but **no ciphertext**. This is what you cast a `@>` operand to. | +| `public.eql_v3_json_search` | The searchable column type. An encrypted document envelope carrying an `sv` array — one encrypted entry per path in the document. | +| `public.eql_v3_json` | Storage-only encrypted JSON: one opaque ciphertext (`{v, i, c}`), no index terms, no server-side searchability. Every comparison and containment operator is blocked on it. | +| `public.eql_v3_json_entry` | A single entry from the vector: a selector, a ciphertext, and an optional ordering term. This is what `->` returns. | +| `eql_v3.query_json` | A containment needle: entries with selectors but **no ciphertext**. This is what you cast a `@>` operand to. | + + +These domains were renamed in EQL 3.0.1: the searchable document was `public.eql_v3_json`, the entry was `public.eql_v3_jsonb_entry`, and the needle was `eql_v3.query_jsonb`. The bare `public.eql_v3_json` name now means the storage-only domain, which rejects a ste_vec document, so retype searchable columns: `ALTER TABLE t ALTER COLUMN doc TYPE public.eql_v3_json_search`. + ## Payload shape -An encrypted JSON document uses a different payload shape from the scalar types: the standard envelope keys are present (`v`, `i`, plus the `k: "sv"` discriminator — envelope anatomy is covered in [Core concepts](/reference/eql/core-concepts)), but there is no root ciphertext. Instead, an `sv` array carries one encrypted entry per path in the document. Each entry has: +An encrypted JSON document uses a different payload shape from the scalar types: the standard envelope keys are present (`v`, `i`, plus the `k: "sv"` discriminator — envelope anatomy is covered in [Core concepts](/reference/eql/core-concepts)), but there is no root ciphertext — the root document's ciphertext is the root `sv` entry. Two document-level keys plus the vector: | Key | Contents | | --- | --- | -| `s` | Selector — a deterministic hash of the JSON path. Required; entry matching compares selectors first. | -| `c` | Ciphertext for the node at that path. | -| `hm` **or** `op` | Exactly one, never both — the domain `CHECK` enforces the exclusivity. `hm` (HMAC-256) on Boolean/`null` leaves and Object/Array roots; `op` (CLLW OPE, backed by `eql_v3_internal.ope_cllw`) on String/Number leaves. | +| `h` | Key header — the key-retrieval material (IV, tag, descriptor, keyset) for the whole document. Required. Hoisted here once rather than repeated in every entry, which shrinks a document by roughly 30%. | +| `sv` | The vector: one encrypted entry per path in the document. | + +Each entry has: + +| Key | Contents | +| --- | --- | +| `s` | Selector — a deterministic hash. On a **path entry** it covers the JSON path; on a **value entry** it covers the path *and* the canonical value, which is what makes exact matching work. Required; entry matching compares selectors first. | +| `c` | Ciphertext for the node at that path — raw AEAD output, with the nonce derived from the entry's own selector rather than a shared per-document IV. A value entry encrypts a fixed versioned sentinel, so it stays distinguishable from a genuine `""` leaf. | +| `op` | Optional ordering term (CLLW OPE, backed by `eql_v3_internal.ope_cllw`) on String and Number path entries. | | `a` | Optional array marker — `true` when the selector points at an array context. | -The decoded `op` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Pre-release payloads named this key `oc` and backed it with CLLW ORE, and older ones split it further, into `ocf` (fixed-width, numeric) and `ocv` (variable-width, string). EQL 3.0.0 ships `op`, backed by CLLW OPE, whose term orders under the default btree operator class. +The decoded `op` value starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, so numeric and string values in one column keep a consistent total order. Pre-release payloads named this key `oc` and backed it with CLLW ORE, and older ones split it further, into `ocf` (fixed-width, numeric) and `ocv` (variable-width, string). -A document payload for a `public.eql_v3_json` column: + +**Entries no longer carry `hm`.** The per-value HMAC equality term was retired in EQL 3.0.1 in favour of value-inclusive selectors, and an `hm`-bearing entry now fails the domain `CHECK`. Along with the key header and the selector-derived nonces, that makes this a wire-format change: stored documents need re-encrypting by a current client, and there is no mechanical conversion. + + +A document payload for a `public.eql_v3_json_search` column: ```json { "v": 3, "k": "sv", "i": { "t": "orders", "c": "metadata" }, + "h": "a4f1c2...", "sv": [ - { "s": "2517068c0d1f9d4d41d2c666211f785e", "c": "mBbKmM...", "hm": "b0e0..." }, + { "s": "2517068c0d1f9d4d41d2c666211f785e", "c": "mBbKmM..." }, { "s": "f510853a4ab9d4f75f51a533ac264c5d", "c": "mBbKmQ...", "op": "01a3f2..." }, { "s": "33743aed3ae636f6bf05cff11ac4b519", "c": "mBbKmR...", "op": "004e19..." } ] } ``` -- First entry: an object root — `hm` only, equality/containment +- First entry: an object root — no ordering term - Second entry: a string leaf — `op` starting with tag `01` - Third entry: a numeric leaf — `op` starting with tag `00` -A containment **query** payload (`eql_v3.query_jsonb`) has the same `sv` shape but its entries carry no `c` — containment matches selectors and index terms, never ciphertexts. This is the needle the client builds for a `@>` query: +A containment **query** payload (`eql_v3.query_json`) has the same `sv` shape but its entries carry no `c` — containment matches selectors, never ciphertexts. This is the needle the client builds for a `@>` query: ```json { @@ -67,44 +84,52 @@ A containment **query** payload (`eql_v3.query_jsonb`) has the same `sv` shape b ## Storing encrypted JSON -Type the column as `public.eql_v3_json`: +Type the column as `public.eql_v3_json_search`: ```sql CREATE TABLE orders ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - metadata public.eql_v3_json + metadata public.eql_v3_json_search ); ``` -There is no database-side configuration step. Which index terms a document carries is decided by the encryption client; typing the column as `public.eql_v3_json` is what makes the encrypted operators and functions resolve. The domain's `CHECK` constraint validates the payload shape on insert, so malformed values are rejected at write time. +There is no database-side configuration step. Which selectors and terms a document carries is decided by the encryption client; typing the column as `public.eql_v3_json_search` is what makes the encrypted operators and functions resolve. The domain's `CHECK` constraint validates the payload shape on insert, so malformed values are rejected at write time. + +For a document you never search, type the column as the storage-only `public.eql_v3_json` instead: it takes the plain `{v, i, c}` envelope, rejects a ste_vec document, and blocks every comparison operator — the JSON analogue of `public.eql_v3_boolean`. Insert and read through the Stack SDK or Proxy, which encrypt the document into the ste_vec payload on write and decrypt it on read. ## What each node type supports -During encryption, the client flattens the document: each unique path gets a deterministic **selector** hash, and each node gets an entry in the `sv` vector carrying index terms for its JSON type: +During encryption, the client flattens the document: each unique path gets a deterministic **selector** hash. Each node then emits a **path entry** (selector over the path, carrying the ordering term where the type has one) and a **value entry**, whose selector covers the path *and* the canonical value: -| JSON node type | Index term | Equality (`=`, `<>`, `GROUP BY`) | Ordering (`<` … `>=`, `MIN`/`MAX`) | -| --- | --- | --- | --- | -| Object | `hm` (HMAC-256) | Yes | No | -| Array | `hm` (HMAC-256) | Yes | No | -| Boolean / JSON `null` | `hm` (HMAC-256) | Yes | No | +| JSON node type | Ordering term | Exact match (`@>` on the value selector) | Ordering (`<` … `>=`, `MIN`/`MAX`) | +| --- | --- | :---: | :---: | +| Object | — | Yes | No | +| Array | — | Yes | No | +| Boolean / JSON `null` | — | Yes | No | | String | `op` (CLLW OPE, string domain) | Yes | Yes | | Number | `op` (CLLW OPE, numeric domain) | Yes | Yes | -Each entry carries exactly one of `hm` or `op` — the domain `CHECK` enforces the exclusivity. `hm` is a deterministic hash, so it supports equality only. `op` is a CLLW OPE term that reveals ordering and, being deterministic, collapses to equality on matching selectors — `eql_v3.eq_term` reads whichever term an entry carries, so equality works uniformly across all node types. +Exact matching is the *presence* of a value selector in the stored document, which makes it injective and loss-free for every type — including `text`, `bigint` and `numeric`, where the old ordering-term comparison collided (`"café"` with `"cafe"`, `9007199254740993` with `…992`). It is expressed as document containment, not as `=` on an extracted leaf: + +```sql +SELECT * FROM orders WHERE metadata @> $1::eql_v3.query_json; +``` JSON `null` here means a `null` literal *inside* the document. A SQL `NULL` column value is not encrypted at all. ## Blocked native jsonb operators -These native PostgreSQL `jsonb` operators are **blocked** on `public.eql_v3_json`. They raise an error rather than silently running plaintext-jsonb semantics against the encrypted payload: +These native PostgreSQL `jsonb` operators are **blocked** on `public.eql_v3_json_search`. They raise an error rather than silently running plaintext-jsonb semantics against the encrypted payload: - Key/path existence: `?`, `?|`, `?&`, `@?`, `@@` - Path extraction: `#>`, `#>>` - Mutation: `-`, `#-`, `||` - Root-document comparison: `=`, `<>`, `<`, `<=`, `>`, `>=` +Equality on the **extract surface** raises too: `col -> 'sel'::text = $1::…` is a blocker for every family, because an extracted path entry carries no value selector and so cannot answer an exact match. Use value-selector containment for that. + Use containment (`@>` / `<@`), field access (`->` / `->>`), or the `eql_v3.jsonb_path_*` functions instead. There is no server-side mutation of an encrypted document — updates re-encrypt in the client. @@ -130,15 +155,15 @@ The examples below all query one encrypted `metadata` document. In plaintext: The `*_selector` placeholders stand for the selector hash of each path: `region_selector` for `$.customer.region`, `total_selector` for `$.total`, and `items_selector` for `$.items`. -### Containment [#fn-contains] +### Containment and exact matching [#fn-contains] -`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle in the client and cast it to `eql_v3.query_jsonb`; `eql_v3.ste_vec_contains(a, b)` is the function form. +`@>` tests whether the encrypted document contains a structure; `<@` is the reverse. Build the needle in the client and cast it to `eql_v3.query_json`; `eql_v3.ste_vec_contains(a, b)` is the function form. This is both the structural-containment operator and the way to match a field exactly — the client emits a value-selector needle for the field and value you're matching. - + ```sql SELECT * FROM orders -WHERE metadata @> $1::eql_v3.query_jsonb; +WHERE metadata @> $1::eql_v3.query_json; ``` @@ -152,9 +177,9 @@ CREATE INDEX orders_metadata_gin ### Field access [#fn-field-access] -`->` returns a `public.eql_v3_jsonb_entry`; `->>` serializes that entry as ciphertext text. Fields are addressed by selector hash, and array elements by 0-based index. +`->` returns a `public.eql_v3_json_entry`; `->>` serializes that entry as ciphertext text. Fields are addressed by selector hash, and array elements by 0-based index. - + ```sql SELECT metadata -> 'selector_hash'::text FROM orders; -- entry @@ -164,42 +189,32 @@ SELECT metadata -> 0 FROM orders; -- array element by inde -The extracted `public.eql_v3_jsonb_entry` is itself comparable. - -### eql_v3.eq_term [#fn-eq-term] - -Equality on an extracted leaf, via `eql_v3.eq_term`. Works on every node type, and drives `GROUP BY` / `DISTINCT`. - - - -```sql -SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) -FROM orders -GROUP BY eql_v3.eq_term(metadata -> 'region_selector'::text); -``` +An extracted `public.eql_v3_json_entry` is ordered-comparable, but not equality-comparable: `=` / `<>` on it raise. - + +`eql_v3.eq_term` has a `public.eql_v3_json_entry` overload, but it is **deprecated** — it is an alias for `eql_v3.ope_term`, returning raw OPE bytes rather than an exact equality term, so it inherits the ordering term's encoding collisions (float rounding on large integers, collation on text). Exact field matching is value-selector containment, above. + ### eql_v3.ord_term [#fn-ord-term] Range comparisons and `ORDER BY` on an extracted **String or Number** leaf, via `eql_v3.ord_term`. - + ```sql SELECT * FROM orders -WHERE (metadata -> 'total_selector'::text) > $1::public.eql_v3_jsonb_entry; +WHERE (metadata -> 'total_selector'::text) > $1::public.eql_v3_json_entry; ``` -A hash index on `eql_v3.eq_term(col -> ''::text)` engages the equality lookup; a btree on `eql_v3.ord_term(...)` engages range and `ORDER BY`. See [Indexes](/reference/eql/indexes). +A btree on `eql_v3.ord_term(col -> ''::text)` engages range and `ORDER BY`; the GIN index on `eql_v3.to_ste_vec_query(col)::jsonb` engages exact matching. See [Indexes](/reference/eql/indexes). ### eql_v3.min / max [#fn-min-max] `MIN` / `MAX` over an extracted ordered leaf. - + ```sql SELECT eql_v3.min(metadata -> 'total_selector'::text) FROM orders; @@ -211,7 +226,7 @@ SELECT eql_v3.min(metadata -> 'total_selector'::text) FROM orders; Path queries take the same selector hashes. `jsonb_path_query` returns every matching entry, `jsonb_path_query_first` the first, and `jsonb_path_exists` a boolean. - + ```sql SELECT eql_v3.jsonb_path_query(metadata, 'selector_hash') FROM orders; -- all matches @@ -223,18 +238,19 @@ SELECT eql_v3.jsonb_path_exists(metadata, 'selector_hash') FROM orders; -- ### eql_v3.jsonb_array_* [#fn-array] -Helpers over an encrypted array node. `jsonb_array_elements` yields encrypted entries; `jsonb_array_elements_text` yields each element as ciphertext text. +Helpers over an encrypted array node. `jsonb_array_elements` yields encrypted entries — each carrying the grafted key header, which makes it the decryptable unit. - + ```sql -SELECT eql_v3.jsonb_array_length(metadata -> 'items_selector'::text) FROM orders; -SELECT eql_v3.jsonb_array_elements(metadata -> 'items_selector'::text) FROM orders; -SELECT eql_v3.jsonb_array_elements_text(metadata -> 'items_selector'::text) FROM orders; +SELECT eql_v3.jsonb_array_length(metadata -> 'items_selector'::text) FROM orders; +SELECT eql_v3.jsonb_array_elements(metadata -> 'items_selector'::text) FROM orders; ``` +`eql_v3.jsonb_array_elements_text`, which streamed each element as bare ciphertext text, was **removed** in EQL 3.0.1: an entry's ciphertext is no longer independently decryptable, so the rows it produced were useless. Use `eql_v3.jsonb_array_elements`. + ## Worked example An `orders` table with an encrypted `metadata` document. The plaintext your application works with: @@ -259,7 +275,7 @@ The client encrypts this into a ste_vec payload with selectors for `$`, `$.custo ```sql CREATE TABLE orders ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - metadata public.eql_v3_json + metadata public.eql_v3_json_search ); INSERT INTO orders (metadata) VALUES ($1); @@ -271,11 +287,11 @@ INSERT INTO orders (metadata) VALUES ($1); ### Query by containment -Find premium orders. The client encrypts the needle `{"customer": {"tier": "premium"}}` into a `ste_vec_query`: +Find premium orders. The client encrypts the needle `{"customer": {"tier": "premium"}}` into a `ste_vec_query` — a value-selector needle, so this is an exact match on the field, not a fuzzy one: ```sql SELECT id FROM orders -WHERE metadata @> $1::eql_v3.query_jsonb; +WHERE metadata @> $1::eql_v3.query_json; ``` Add the GIN index from above once the table grows. @@ -285,13 +301,12 @@ Add the GIN index from above once the table grows. ### Query by path -Count orders per region, grouping on the encrypted leaf — the database never sees `"apac"`: +Path existence and range comparisons work on the extract surface — the database never sees `"apac"` or `149.95`: ```sql -SELECT eql_v3.eq_term(metadata -> 'region_selector'::text) AS region, COUNT(*) -FROM orders +SELECT id FROM orders WHERE eql_v3.jsonb_path_exists(metadata, 'region_selector') -GROUP BY 1; + AND (metadata -> 'total_selector'::text) > $1::public.eql_v3_json_entry; ``` The rows come back as ciphertext; decrypt them in the client. diff --git a/content/docs/reference/eql/text.mdx b/content/docs/reference/eql/text.mdx index af48ec4..c82edb9 100644 --- a/content/docs/reference/eql/text.mdx +++ b/content/docs/reference/eql/text.mdx @@ -1,10 +1,10 @@ --- title: Text -description: "The complete reference for encrypted text columns: every text domain variant, the multi-term payload, why LIKE is gone everywhere, and bloom-filter token containment as the encrypted free-text match." +description: "The complete reference for encrypted text columns: every text domain variant, the multi-term payload, why LIKE is gone everywhere, and @@ bloom-filter fuzzy match as the encrypted free-text search." type: reference components: [eql] verifiedAgainst: - eql: "3.0.0" + eql: "3.0.3" --- @@ -22,8 +22,8 @@ All of these are `jsonb`-backed domains. Which one you declare fixes the column' | `public.eql_v3_text_ord` | Comparisons, `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | | `public.eql_v3_text_ord_ope` | The byte-identical twin of `text_ord`, with OPE pinned. See [SEM specifiers](#sem-specifiers). | | `public.eql_v3_text_ord_ore` | As `text_ord`, with the ORE mechanism pinned. | -| `public.eql_v3_text_match` | Free-text token containment: `@>` / `<@`. | -| `public.eql_v3_text_search` | Equality + ordering + token containment. | +| `public.eql_v3_text_match` | Free-text fuzzy match: `@@`. | +| `public.eql_v3_text_search` | Equality + ordering + fuzzy match. | | `public.eql_v3_text_search_ore` | As `text_search`, with the ORE mechanism pinned. | Declare only the capabilities you query on — each capability stores extra searchable material with defined leakage (see [Searchable encryption](/concepts/searchable-encryption)). @@ -77,7 +77,7 @@ A value for a `text_search` column carries the shared envelope keys (`v`, `i`, ` - `hm` — equality term: `WHERE email = $1` compares this - `op` — ordering term: a hex-encoded CLLW OPE ciphertext, which `ORDER BY` and range comparisons sort by native `bytea` comparison -- `bf` — bloom-filter term: `@>` token containment tests these bit positions +- `bf` — bloom-filter term: `@@` fuzzy match tests these bit positions A `text_search_ore` payload carries `ob` in place of `op`: an array of block-ORE ciphertexts rather than a single string. @@ -91,7 +91,8 @@ The narrower variants carry only their own terms. A `text_eq` payload carries `h | --- | :---: | :---: | :---: | :---: | :---: | | `=` / `<>` | ❌ | ✅ | ✅ | ❌ | ✅ | | `<` `<=` `>` `>=` | ❌ | ❌ | ✅ | ❌ | ✅ | -| `@>` / `<@` | ❌ | ❌ | ❌ | ✅ | ✅ | +| `@@` (fuzzy match) | ❌ | ❌ | ❌ | ✅ | ✅ | +| `@>` / `<@` | ❌ | ❌ | ❌ | ❌ | ❌ | | `LIKE` / `ILIKE` (`~~` / `~~*`) | ❌ | ❌ | ❌ | ❌ | ❌ | | `IN` / `GROUP BY` / `DISTINCT` | ❌ | ✅ | ✅ | ❌ | ✅ | | `ORDER BY` | ❌ | ❌ | ✅ | ❌ | ✅ | @@ -103,21 +104,46 @@ Blocked *operator* cells raise an `operator … is not supported` exception — content/partials/eql/functions-text.mdx -There are no `like` / `ilike` function forms — encrypted text matching is `eql_v3.contains` on a `text_match` value. +There are no `like` / `ilike` function forms — encrypted text matching is `eql_v3.matches` on a `text_match` value. ## There is no `LIKE` -`LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant — including `text_match` and `text_search`. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment — `@>` on a `text_match` or `text_search` column: +`LIKE` and `ILIKE` (`~~` / `~~*`) raise on **every** encrypted-domain variant — including `text_match` and `text_search`. SQL pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter fuzzy match — `@@` on a `text_match` or `text_search` column: + + ```sql --- ❌ Raises: operator not supported SELECT * FROM users WHERE email LIKE '%alice%'; +``` + + --- ✅ Encrypted free-text match -SELECT * FROM users WHERE email @> $1::public.eql_v3_text_match; +```sql +-- Encrypted free-text match +SELECT * FROM users WHERE email @@ $1::public.eql_v3_text_match; + +-- Function form +SELECT * FROM users WHERE eql_v3.matches(email, $1::public.eql_v3_text_match); ``` -`@>` / `<@` here is **probabilistic ngram-bloom containment** — it tests whether the encrypted text contains the (encrypted) search terms. It is not JSONB containment and not `LIKE`. The client encrypts the search term into a bloom-filter query value; false positives are possible, false negatives are not. There are no `like` / `ilike` function forms either — text matching is `eql_v3.contains` on a `text_match` value. +`@@` is **probabilistic ngram-bloom matching** — it reduces to array containment over two bloom filters built from the downcased 3-gram token sets, so a `true` may be a false positive and a `false` never is. It is not `LIKE`, and it is not containment: it is order- and multiplicity-insensitive. The client encrypts the search term into a bloom-filter query value. + + +**`@>` / `<@` raise on the match variants.** The fuzzy match was renamed in EQL 3.0.1: it used to be spelled `@>` / `<@`, backed by `eql_v3.contains` / `eql_v3.contained_by`, which are no longer public functions on these domains. That naming promised containment semantics it never had, so the operator became the directional `@@` and the old spellings are now blockers that raise `operator … is not supported`. Genuine containment on encrypted JSON is unaffected and keeps `@>` / `<@`; see [JSON](/reference/eql/json). Repoint any query, view, or adapter still emitting the old form, and coordinate with the client: an SDK emitting `eql_v3.contains(` for text match must move to `eql_v3.matches(` in the same release. + + +### Search terms too short to tokenise + +A search string below the tokeniser's floor — a 2-character term, say — produces no 3-grams and encrypts to an empty bloom (`bf: []`). Because an empty array is contained by every array, such a needle once matched **every row in the table**, silently. Since EQL 3.0.3 the match is guarded, giving `LIKE ''`-shaped semantics: + +| Stored value's bloom | Needle's bloom | `LIKE` analogue | Result | +| --- | --- | --- | :---: | +| empty | empty | `'' LIKE ''` | ✅ | +| non-empty | empty | `'catty' LIKE ''` | ❌ (was ✅) | +| empty | non-empty | `'' LIKE 'cat'` | ❌ | +| non-empty | non-empty | — | bloom match | + +Only the second row changed: an empty needle now matches only a value whose own bloom is also empty. Nothing to rewrite, and values legitimately stored with an empty bloom are unaffected. On EQL 3.0.2 and earlier, reject sub-floor search terms client-side — the SDK already does. ## Where to next diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index c895923..70fb045 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -283,7 +283,16 @@ interface FuncDef { id: string; name: string; ops: string[]; - cap: string; + /** Domain capability that exposes the function. */ + cap?: string; + /** + * Operator that exposes the function, for surfaces the catalog does not model + * as a capability. Fuzzy match is the case: 3.0.1 dropped the `match` + * capability from the catalog, and `public.eql_v3_text_match` now reports + * `capabilities: ["storage"]` with `supportedOperators: ["@@"]`, so the + * operator list is the only reliable signal that a domain can be matched on. + */ + op?: string; agg?: boolean; } const FUNCS: FuncDef[] = [ @@ -296,16 +305,10 @@ const FUNCS: FuncDef[] = [ cap: "order", }, { - id: "fn-contains", - name: "eql_v3.contains(a, b)", - ops: ["@>"], - cap: "match", - }, - { - id: "fn-contained_by", - name: "eql_v3.contained_by(a, b)", - ops: ["<@"], - cap: "match", + id: "fn-matches", + name: "eql_v3.matches(a, b)", + ops: ["@@"], + op: "@@", }, { id: "fn-min", @@ -341,10 +344,8 @@ function exampleFor(id: string, spec: FragmentSpec): string { return spec.orderByExample ? `-- any of the four; ordering is the usual reason to index text\nSELECT id, ${col} FROM ${table}\nWHERE eql_v3.gt(${col}, $1::${dom("ord")})\nORDER BY eql_v3.ord_term(${col});` : `-- a range uses two of the four\nSELECT * FROM ${table}\nWHERE eql_v3.gte(${col}, $1::${dom("ord")})\n AND eql_v3.lt(${col}, $2::${dom("ord")});`; - case "fn-contains": - return `-- token containment on the bloom-filter term\nSELECT * FROM ${table}\nWHERE eql_v3.contains(${col}, $1::${dom("match")});`; - case "fn-contained_by": - return `SELECT * FROM ${table}\nWHERE eql_v3.contained_by(${col}, $1::${dom("match")});`; + case "fn-matches": + return `-- probabilistic fuzzy match on the bloom-filter term\nSELECT * FROM ${table}\nWHERE eql_v3.matches(${col}, $1::${dom("match")});`; case "fn-min": return `-- compares ordering terms; result decrypts client-side\nSELECT eql_v3.min(${col}) FROM ${table};`; case "fn-max": @@ -366,7 +367,12 @@ function renderFragment(domains: Domain[], spec: FragmentSpec): string { const blocks: string[] = []; for (const fn of FUNCS) { const applies = scoped - .filter((d) => d.capabilities.includes(fn.cap) && d.variant) + .filter((d) => { + if (!d.variant) return false; + return fn.op + ? (d.supportedOperators ?? []).includes(fn.op) + : !!fn.cap && d.capabilities.includes(fn.cap); + }) .map((d) => shortDomain(d.name)); if (!applies.length) continue; const attrs = [ @@ -417,6 +423,11 @@ function knownSymbols(manifest: Manifest): Set { return known; } +// Prose that names a symbol in order to say it no longer exists (or no longer +// means what it used to). Kept narrow — these are the words the upgrade notes +// and rename callouts actually use, not a general escape hatch. +const RETIRED_MARKER = /\b(removed|renamed|retired|deprecated|no longer)\b/i; + function driftCheck(manifest: Manifest): string[] { const known = knownSymbols(manifest); const referenced = new Map>(); // fqn -> pages @@ -424,26 +435,33 @@ function driftCheck(manifest: Manifest): string[] { for (const file of fs.readdirSync(EQL_DIR)) { if (!file.endsWith(".mdx") || file === "functions.mdx") continue; const text = fs.readFileSync(path.join(EQL_DIR, file), "utf8"); - // Any schema-qualified reference — function call, domain cast, or type. - // A trailing `*` marks a prose family (e.g. `eql_v3.jsonb_path_*`), which - // names a set rather than one symbol, so it's skipped. So is a trailing - // `` placeholder (e.g. `public.eql_v3__ord`). - for (const m of text.matchAll( - /\b(public|eql_v3_internal|eql_v3)\.([a-z0-9_]+)(\*?)/g, - )) { - if (m[3] === "*") continue; - if (text[(m.index ?? 0) + m[0].length] === "<") continue; - const fqn = `${m[1]}.${m[2]}`; - const pages = referenced.get(fqn) ?? new Set(); - pages.add(file); - referenced.set(fqn, pages); + for (const line of text.split("\n")) { + // A line that names a symbol in order to say it is gone is documenting + // history, not teaching an API — an upgrade note has to be able to write + // "`eql_v3.jsonb_array_elements_text` was removed". Same line-local + // exemption validate-content-api.ts makes, and the same trade: naming a + // dead symbol WITHOUT saying it is dead is the bug being caught. + if (RETIRED_MARKER.test(line)) continue; + // Any schema-qualified reference — function call, domain cast, or type. + // A trailing `*` marks a prose family (e.g. `eql_v3.jsonb_path_*`), which + // names a set rather than one symbol, so it's skipped. So is a trailing + // `` placeholder (e.g. `public.eql_v3__ord`). + for (const m of line.matchAll( + /\b(public|eql_v3_internal|eql_v3)\.([a-z0-9_]+)(\*?)/g, + )) { + if (m[3] === "*") continue; + if (line[(m.index ?? 0) + m[0].length] === "<") continue; + const fqn = `${m[1]}.${m[2]}`; + const pages = referenced.get(fqn) ?? new Set(); + pages.add(file); + referenced.set(fqn, pages); + } } } const unknown: string[] = []; for (const [fqn, pages] of referenced) { if (known.has(fqn) || MANIFEST_BLIND_SPOTS.has(fqn)) continue; - if (UNRELEASED.has(fqn)) continue; unknown.push(`${fqn} (in ${[...pages].join(", ")})`); } return unknown.sort(); @@ -460,22 +478,23 @@ function driftCheck(manifest: Manifest): string[] { // // Keep this list minimal, and delete entries as the manifest grows to cover // them. -const MANIFEST_BLIND_SPOTS = new Set(["eql_v3.query_text_eq"]); - -// Symbols merged into EQL but not yet in the pinned release, documented ahead of -// it because a page would otherwise teach a workaround for a solved problem. The -// manifest can't resolve them, so they're allowlisted here — and reported on -// every run, so the list stays visible rather than becoming a quiet exemption. -// -// The page documenting one MUST carry a version callout saying which release it -// needs. Delete the entry when EQL_RELEASE_TAG (scripts/generate-eql-docs.ts) is -// bumped to a release that ships it: the manifest covers it from then on, and -// the drift check goes back to being the authority. -const UNRELEASED = new Map([ - [ - "eql_v3.grouped_value", - "added after eql-3.0.2 in cipherstash/encrypt-query-language#423; documented in grouping-and-aggregates.mdx", - ], +// `eql_v3.grouped_value` is the second blind spot, with a different cause: the +// manifest lists aggregates from the codegen catalog, so the 60 generated +// `min`/`max` aggregates are there but the one hand-written `CREATE AGGREGATE` +// is not. Verified present in the 3.0.3 install SQL: +// grep -c "CREATE AGGREGATE" cipherstash-encrypt.sql # => 61 +// grep -c "CREATE AGGREGATE eql_v3.grouped_value" cipherstash-encrypt.sql # => 1 +// Its state function (`eql_v3_internal.grouped_value_sfunc`) IS in the manifest, +// which is what makes the gap easy to miss. +// `public.eql_v3_json` — the storage-only JSON domain added in 3.0.1 — has the +// same cause as the query-operand domains: created in a `DO` block, so the +// catalog the manifest is built from never sees it (its searchable sibling +// `public.eql_v3_json_search` IS in the manifest). Verified in the 3.0.3 SQL: +// grep -c "CREATE DOMAIN public.eql_v3_json AS jsonb" cipherstash-encrypt.sql # => 1 +const MANIFEST_BLIND_SPOTS = new Set([ + "eql_v3.query_text_eq", + "eql_v3.grouped_value", + "public.eql_v3_json", ]); // ── Main ───────────────────────────────────────────────────────────────────── @@ -499,10 +518,6 @@ function main() { `✓ Generated ${path.relative(process.cwd(), OUT_FILE)} from EQL ${manifest.version} (${manifest.functions.length} functions)`, ); - for (const [fqn, why] of UNRELEASED) { - console.log(`• Drift check: ${fqn} allowlisted as unreleased — ${why}`); - } - const unknown = driftCheck(manifest); if (unknown.length) { const header = `⚠ ${unknown.length} schema-qualified symbol(s) referenced in hand-written pages are not in the manifest (domains or functions):`; diff --git a/scripts/generate-eql-docs.ts b/scripts/generate-eql-docs.ts index 3b72377..d934247 100644 --- a/scripts/generate-eql-docs.ts +++ b/scripts/generate-eql-docs.ts @@ -24,11 +24,13 @@ import path from "node:path"; import GithubSlugger from "github-slugger"; /** - * The EQL release the docs are written against. EQL 3.0.0 is still in - * prerelease and churning (alpha.3 and alpha.4 shipped a day apart), so this - * tracks an alpha deliberately rather than by accident. + * The EQL release the docs are written against. + * + * 3.0.x is still moving: 3.0.1 renamed the encrypted-JSON domains and replaced + * the text fuzzy-match operator, 3.0.3 guarded the empty-bloom needle. Each bump + * is a deliberate commit that fixes whatever the drift check reports. */ -const EQL_RELEASE_TAG = process.env.EQL_RELEASE_TAG ?? "eql-3.0.0"; +const EQL_RELEASE_TAG = process.env.EQL_RELEASE_TAG ?? "eql-3.0.3"; const GITHUB_RELEASE_DOWNLOAD = "https://github.com/cipherstash/encrypt-query-language/releases/download"; From a1538d3c5a2e1c0d923962a4404f47e8a64d0bbb Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 28 Jul 2026 09:49:53 +1000 Subject: [PATCH 4/5] docs: escape manifest prose before it reaches MDX MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EQL 3.0.3's ope_term brief contains "= / <> are blocked". Doxygen comments are written for a plain-text renderer, so bare SQL operators are normal in them — but the generator dropped them into functions.mdx verbatim, where MDX read `<>` as an unclosed JSX fragment and failed the build on the generated content file rather than at the source. Escape `<` and `{` in every prose string the manifest supplies (brief, description, param and return descriptions), skipping inline-code spans so the descriptions that already write `<>` in backticks keep rendering as code. --- scripts/generate-eql-api-docs.ts | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index 70fb045..e78e60e 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -93,12 +93,31 @@ function loadManifest(): Manifest { } // ── Render the generated catalog ───────────────────────────────────────────── +/** + * Make a manifest prose string safe to drop into MDX. + * + * Doxygen comments are written for a plain-text renderer, so they contain bare + * `<` and `{` — SQL operators especially (`<>`, `<=`). MDX reads those as JSX + * and the build dies on the *content* file, far from the cause: EQL 3.0.3's + * `ope_term` brief says "= / <> are blocked", which parses as an unclosed + * fragment. Escape both, but only outside inline-code spans, so the many + * descriptions that already write `` `<>` `` keep rendering as code. + */ +function mdxProse(s: string): string { + return s + .split(/(`+[^`]*`+)/g) + .map((seg, i) => + i % 2 === 1 ? seg : seg.replace(/ - `| \`${p.name}\` | ${p.type ? `\`${p.type}\`` : ""} | ${(p.description ?? "").replace(/\|/g, "\\|")} |`, + `| \`${p.name}\` | ${p.type ? `\`${p.type}\`` : ""} | ${mdxProse(p.description ?? "").replace(/\|/g, "\\|")} |`, ) .join("\n"); return `\n| Parameter | Type | Description |\n| --- | --- | --- |\n${rows}\n`; @@ -123,9 +142,9 @@ function renderPublicFunctions(fns: Fn[]): string { const rep = overloads.find((o) => o.params.length || o.brief) ?? overloads[0]; const sigs = [...new Set(overloads.map((o) => o.signature))].sort(); - const parts = [`### \`${name}\``, "", rep.brief]; + const parts = [`### \`${name}\``, "", mdxProse(rep.brief)]; if (rep.description && rep.description !== rep.brief) - parts.push("", rep.description); + parts.push("", mdxProse(rep.description)); parts.push( "", sigs.length > 1 @@ -141,7 +160,7 @@ function renderPublicFunctions(fns: Fn[]): string { const t = rep.returns.type ? `\`${rep.returns.type}\`` : ""; parts.push( "", - `**Returns:** ${t}${rep.returns.description ? ` — ${rep.returns.description}` : ""}`, + `**Returns:** ${t}${rep.returns.description ? ` — ${mdxProse(rep.returns.description)}` : ""}`, ); } sections.push(parts.join("\n")); From 90916c45bec17e64e830d0a9d08db7f6eb141708 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 28 Jul 2026 17:34:46 +1000 Subject: [PATCH 5/5] docs: pin EQL 3.0.4 and drop the two manifest blind spots it fixed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit EQL 3.0.4 ships the manifest-extraction fixes from cipherstash/encrypt-query-language#427, so `eql_v3.grouped_value` and `public.eql_v3_json` now resolve from the catalog and no longer need drift-check exemptions. Verified against the released manifest: grouped_value(jsonb) and lints() present, 53 domains (was 52) including public.eql_v3_json, and the `match` capability restored to text_match and both text_search variants. Only `eql_v3.query_text_eq` stays on the allowlist — the 40 scalar query-operand domains created in a `DO ... EXECUTE` block are a separate gap that 3.0.4 does not address. Removing the other two means a regression in either now fails the drift check instead of staying quiet. Also always stamp the pinned tag as the manifest's version rather than only substituting when it reads "DEV". EQL's release workflow derives that field from a tag input that can arrive empty, in which case json.sh falls back to "DEV" — which 3.0.4 shipped, and which would otherwise render as "EQL DEV" in the banner on every reference page. The existing guard already covered that case; this extends it to warn when a manifest is stamped with a DIFFERENT release, which would be a packaging mix-up worth seeing rather than a cosmetic default worth silencing. bun run build passes: drift check clean against 3.0.4, 1990 functions, banner reads EQL 3.0.4. Claude-Session: https://claude.ai/code/session_01NkuQNMvw9BpB4BWWeKtfV8 --- scripts/generate-eql-api-docs.ts | 28 ++++++++++------------------ scripts/generate-eql-docs.ts | 30 ++++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/scripts/generate-eql-api-docs.ts b/scripts/generate-eql-api-docs.ts index e78e60e..5bcc536 100644 --- a/scripts/generate-eql-api-docs.ts +++ b/scripts/generate-eql-api-docs.ts @@ -497,24 +497,16 @@ function driftCheck(manifest: Manifest): string[] { // // Keep this list minimal, and delete entries as the manifest grows to cover // them. -// `eql_v3.grouped_value` is the second blind spot, with a different cause: the -// manifest lists aggregates from the codegen catalog, so the 60 generated -// `min`/`max` aggregates are there but the one hand-written `CREATE AGGREGATE` -// is not. Verified present in the 3.0.3 install SQL: -// grep -c "CREATE AGGREGATE" cipherstash-encrypt.sql # => 61 -// grep -c "CREATE AGGREGATE eql_v3.grouped_value" cipherstash-encrypt.sql # => 1 -// Its state function (`eql_v3_internal.grouped_value_sfunc`) IS in the manifest, -// which is what makes the gap easy to miss. -// `public.eql_v3_json` — the storage-only JSON domain added in 3.0.1 — has the -// same cause as the query-operand domains: created in a `DO` block, so the -// catalog the manifest is built from never sees it (its searchable sibling -// `public.eql_v3_json_search` IS in the manifest). Verified in the 3.0.3 SQL: -// grep -c "CREATE DOMAIN public.eql_v3_json AS jsonb" cipherstash-encrypt.sql # => 1 -const MANIFEST_BLIND_SPOTS = new Set([ - "eql_v3.query_text_eq", - "eql_v3.grouped_value", - "public.eql_v3_json", -]); +// +// `eql_v3.grouped_value` and `public.eql_v3_json` used to be listed here too. +// Both were fixed upstream in EQL 3.0.4 +// (cipherstash/encrypt-query-language#427): the aggregate was lost because +// Doxygen misread `CREATE AGGREGATE`'s definition body and named the member +// after its argument type, and the storage-only JSON domain was filtered out of +// the catalog dump as scalar while the scalar path skipped its mixed family +// wholesale. Both now resolve from the manifest, so the exemptions are gone — +// and if either regresses, the drift check says so instead of staying quiet. +const MANIFEST_BLIND_SPOTS = new Set(["eql_v3.query_text_eq"]); // ── Main ───────────────────────────────────────────────────────────────────── function main() { diff --git a/scripts/generate-eql-docs.ts b/scripts/generate-eql-docs.ts index d934247..6f9b171 100644 --- a/scripts/generate-eql-docs.ts +++ b/scripts/generate-eql-docs.ts @@ -27,10 +27,13 @@ import GithubSlugger from "github-slugger"; * The EQL release the docs are written against. * * 3.0.x is still moving: 3.0.1 renamed the encrypted-JSON domains and replaced - * the text fuzzy-match operator, 3.0.3 guarded the empty-bloom needle. Each bump - * is a deliberate commit that fixes whatever the drift check reports. + * the text fuzzy-match operator, 3.0.3 guarded the empty-bloom needle, and + * 3.0.4 fixed the manifest extraction that had been dropping + * `eql_v3.grouped_value` and `public.eql_v3_json` from the catalog entirely + * (cipherstash/encrypt-query-language#427). Each bump is a deliberate commit + * that fixes whatever the drift check reports. */ -const EQL_RELEASE_TAG = process.env.EQL_RELEASE_TAG ?? "eql-3.0.3"; +const EQL_RELEASE_TAG = process.env.EQL_RELEASE_TAG ?? "eql-3.0.4"; const GITHUB_RELEASE_DOWNLOAD = "https://github.com/cipherstash/encrypt-query-language/releases/download"; @@ -278,13 +281,24 @@ async function main() { // this is best-effort and the generator falls back to the committed sample. const manifestSrc = path.join(extractPath, "json", "eql-manifest.json"); try { - // Prerelease manifests report `"version": "DEV"`, which would surface as - // "generated and validated against EQL DEV" in the banner on every - // reference page. We know which release we pinned, so say so. + // The pinned tag is the authoritative version, so always stamp it rather + // than trusting the manifest's own field. EQL's release workflow derives + // that field from a tag input that can arrive empty, in which case + // `json.sh` silently falls back to `"DEV"` — which eql-3.0.4 shipped, and + // which would otherwise surface as "EQL DEV" in the banner on every + // reference page. Report a disagreement rather than papering over it: a + // manifest stamped with a DIFFERENT release is a packaging mix-up worth + // seeing, not a cosmetic default. const manifest = JSON.parse(await fs.readFile(manifestSrc, "utf8")); - if (!manifest.version || manifest.version === "DEV") { - manifest.version = tag.replace(/^eql-/, ""); + const expected = tag.replace(/^eql-/, ""); + if (manifest.version && manifest.version !== "DEV") { + if (manifest.version !== expected) { + console.warn( + `⚠ ${tag} ships a manifest stamped "${manifest.version}", not "${expected}". Using the pinned tag; check the release.`, + ); + } } + manifest.version = expected; await fs.writeFile(RELEASE_MANIFEST, JSON.stringify(manifest, null, 2)); console.log( `✓ Extracted eql-manifest.json → ${path.basename(RELEASE_MANIFEST)} (version: ${manifest.version})`,