docs: EQL 3.0.4 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface - #74
Conversation
…hapes EQL #423 added `eql_v3.grouped_value`, the aggregate that lets a query project an encrypted column while grouping by that column's equality term. Documenting it exposed two wrong claims on the grouping page: that `GROUP BY email` and `SELECT DISTINCT email` work in natural form, and that grouping compares equality terms. Neither is true — an encrypted column is a domain over jsonb with no custom operator class, so Postgres compares the raw (randomised) ciphertext and never collapses two encryptions of the same value. - Group on `eql_v3.eq_term(col)` for correctness first, planner economics second; drop the "raise work_mem if an ORM forces the raw form" advice, which rescued the performance of a query that was already returning wrong answers. - New `grouped_value` section: the 42803 rejection it fixes, the shape that fixes it, and its semantics (first non-null per group, all-NULL group is NULL, parallel safe, returns its input unchanged). - Deduplication is `DISTINCT ON (eql_v3.eq_term(col)) col` with the matching leading ORDER BY. Plain DISTINCT is not supported; `DISTINCT ON` has no projection restriction, so `grouped_value` is not involved there. - Same correction to the Indexes page's GROUP BY section, and grouped_value in the MIN/MAX and JSON-leaf examples. grouped_value landed after eql-3.0.2, so the page carries a version callout (with the 3.0.x join-back workaround) and the drift check gets an UNRELEASED allowlist — reported on every run, and deleted when EQL_RELEASE_TAG is bumped to a release whose manifest covers it.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…rsion note - Restore the `GROUP BY` and `DISTINCT` heading, and fold deduplication back under it: one section for how both operators must be written, with the `grouped_value` projection following as its own section. - Add a `BadExample` component (rustdoc `compile_fail` styling: red frame, explicit label, verbatim error) so a broken example can't be mistaken for a working one while skimming, and use it for the two broken shapes here — the natural `GROUP BY` / `DISTINCT` forms, and the bare column under GROUP BY. - Drop the unreleased-version callout.
Documenting U-011 (the empty-bloom fuzzy-match guard, shipped in 3.0.3) meant moving EQL_RELEASE_TAG off 3.0.0, and the strict drift check then reported what the pin had been hiding: 3.0.1 renamed the whole encrypted-JSON surface and replaced the text match operator, and the hand-written pages still taught the 3.0.0 spelling of both. Every customer on a released 3.0.x has been reading docs whose text-match and JSON examples raise. Text fuzzy match (U-008, U-011): - `@>` / `<@` and `eql_v3.contains` / `contained_by` on text_match, text_search and text_search_ore are now `@@` / `eql_v3.matches`; the old spellings raise, and the pages say so. - Document the empty-needle guard as a LIKE truth table: a sub-floor search term (no trigrams, `bf: []`) used to match every row and now matches only empty-bloom values. - Note the one regression it carries: `matches` is non-STRICT and names the needle twice, so a needle from an uncorrelated subquery stops inlining and loses the GIN index. Bind parameters and literals are unaffected. Encrypted JSON (U-007, U-010): - eql_v3_json → eql_v3_json_search, eql_v3_jsonb_entry → eql_v3_json_entry, query_jsonb → query_json; the bare eql_v3_json is now storage-only. - Field equality is value-selector containment, not `=` on an extracted leaf (which raises). `eq_term` on an entry is a deprecated alias for `ope_term` and is not an exact equality term — so JSON leaves can no longer be grouped by equality at all, which the grouping page now states outright. - Wire format: document-level key header `h`, no root `c`, selector-derived nonces, sentinel value entries, `hm` retired. - `jsonb_array_elements_text` was removed; entry ciphertext is not independently decryptable. Tooling: - The fragment generator keyed the match card off a `match` capability the 3.0.1 catalog no longer emits (text_match now reports `["storage"]` with `supportedOperators: ["@@"]`), so the text function catalog silently lost its match entry. It now keys off the operator and renders `eql_v3.matches`. - Drift check: exempt lines that name a symbol in order to say it is gone, so a rename note can name the old symbol — the same line-local exemption validate-content-api.ts makes. - Two more manifest blind spots, both verified present in the 3.0.3 install SQL: `eql_v3.grouped_value` (hand-written CREATE AGGREGATE; the catalog only contributes the 60 generated min/max aggregates) and `public.eql_v3_json` (created in a DO block, like the query-operand domains). Generated artifacts (functions.mdx, the partials, eql-version.ts, the stack EQL API page) are left alone: prebuild regenerates them from the pinned release, and the committed copies are placeholders — the stack one still says EQL 2.3.0-pre.3.
EQL 3.0.3's ope_term brief contains "= / <> are blocked". Doxygen comments
are written for a plain-text renderer, so bare SQL operators are normal in
them — but the generator dropped them into functions.mdx verbatim, where MDX
read `<>` as an unclosed JSX fragment and failed the build on the generated
content file rather than at the source.
Escape `<` and `{` in every prose string the manifest supplies (brief,
description, param and return descriptions), skipping inline-code spans so
the descriptions that already write `<>` in backticks keep rendering as code.
EQL 3.0.4 ships the manifest-extraction fixes from cipherstash/encrypt-query-language#427, so `eql_v3.grouped_value` and `public.eql_v3_json` now resolve from the catalog and no longer need drift-check exemptions. Verified against the released manifest: grouped_value(jsonb) and lints() present, 53 domains (was 52) including public.eql_v3_json, and the `match` capability restored to text_match and both text_search variants. Only `eql_v3.query_text_eq` stays on the allowlist — the 40 scalar query-operand domains created in a `DO ... EXECUTE` block are a separate gap that 3.0.4 does not address. Removing the other two means a regression in either now fails the drift check instead of staying quiet. Also always stamp the pinned tag as the manifest's version rather than only substituting when it reads "DEV". EQL's release workflow derives that field from a tag input that can arrive empty, in which case json.sh falls back to "DEV" — which 3.0.4 shipped, and which would otherwise render as "EQL DEV" in the banner on every reference page. The existing guard already covered that case; this extends it to warn when a manifest is stamped with a DIFFERENT release, which would be a packaging mix-up worth seeing rather than a cosmetic default worth silencing. bun run build passes: drift check clean against 3.0.4, 1990 functions, banner reads EQL 3.0.4. Claude-Session: https://claude.ai/code/session_01NkuQNMvw9BpB4BWWeKtfV8
|
Rebased on current
-const MANIFEST_BLIND_SPOTS = new Set([
- "eql_v3.query_text_eq",
- "eql_v3.grouped_value",
- "public.eql_v3_json",
-]);
+const MANIFEST_BLIND_SPOTS = new Set(["eql_v3.query_text_eq"]);Verified against the released manifest: One extra change: the pinned tag is now always stamped as the manifest's version, not only when it reads
|
Started as encrypt-query-language#423 (
eql_v3.grouped_value), and grew into the EQL 3.0.3 pin bump once 3.0.3 shipped. Three related things, all driven by the same root cause: the docs were pinned to 3.0.0, and 3.0.1–3.0.3 changed surfaces the hand-written pages teach.1.
eql_v3.grouped_valueand the GROUP BY / DISTINCT shapesgrouped_valueprojects an encrypted column while grouping by its equality term. Writing it up surfaced two wrong claims: thatGROUP BY emailandSELECT DISTINCT emailwork in natural form, and that grouping compares equality terms. Neither holds — an encrypted column is a domain overjsonbwith no operator class of its own, so Postgres compares randomised ciphertext and every row lands in its own group, silently.eql_v3.eq_term(col)for correctness first, planner economics second. The old "raisework_memif an ORM forces the raw form" advice is gone — it tuned a query that was already wrong.eql_v3.grouped_value: the 42803 rejection it fixes, the shape that fixes it, and its semantics (first non-NULLper group, all-NULL→NULL,PARALLEL SAFE, input returned unchanged so it decrypts client-side).DISTINCT ON (eql_v3.eq_term(col)) colwith a matching leadingORDER BY; plainDISTINCTisn't supported, andDISTINCT ONhas no projection restriction, sogrouped_valueisn't involved there.2. Text fuzzy match:
@@/eql_v3.matches(U-008, U-011)3.0.1 replaced
@>/<@andeql_v3.contains/contained_byontext_match/text_search/text_search_orewith the directional@@/eql_v3.matches; the old spellings now raise. The docs still taught them — ontext.mdx,filtering.mdx,core-concepts.mdxandindexes.mdx.3.0.3 then added the U-011 guard, documented here as a
LIKEtruth table: a search term below the tokeniser floor (no trigrams,bf: []) used to match every row in the table, and now matches only empty-bloom values. The one regression it carries is documented too —matchesis non-STRICTand names the needle twice, so a needle from an uncorrelated subquery stops inlining and loses the GIN index; bind parameters and literals are unaffected.3. The encrypted-JSON surface (U-007, U-010)
Not in the original scope, but the pin bump made it a build gate — and these pages have been wrong for every reader on a released 3.0.x:
eql_v3_json→eql_v3_json_search,eql_v3_jsonb_entry→eql_v3_json_entry,query_jsonb→query_json. The bareeql_v3_jsonis now a storage-only domain.=on an extracted leaf, which raises.eq_termon an entry is a deprecated alias forope_termreturning OPE bytes — so extracted JSON leaves can no longer be grouped by equality at all, which the grouping page now states outright instead of showing an example that doesn't work.h, no rootc, selector-derived nonces, sentinel value entries,hmretired — a re-encryption, not a rename.jsonb_array_elements_textwas removed; entry ciphertext isn't independently decryptable.Tooling
matchcapability the 3.0.1 catalog no longer emits (text_matchreportscapabilities: ["storage"]withsupportedOperators: ["@@"]), so the text function reference had quietly lost its match entry. It now keys off the operator and renderseql_v3.matches.validate-content-api.ts, so a rename note can name the old symbol.eql_v3.grouped_value(hand-writtenCREATE AGGREGATE— the catalog only contributes the 60 generatedmin/maxaggregates, so 61 ship and 60 are listed) andpublic.eql_v3_json(created in aDOblock, same cause as the query-operand domains). Both look like upstream manifest gaps worth fixing in the EQL repo — the generated API reference is missing them entirely.BadExamplecomponent (src/components/bad-example.tsx), rustdoccompile_fail-style: red frame, an explicit label distinguishing "Raises" from "Silently wrong", and the verbatim error. Used for the four broken examples on these pages.Generated artifacts (
functions.mdx, the partials,eql-version.ts, the stack EQL API page) are deliberately not committed: prebuild regenerates them from the pinned release, and the committed copies are placeholders — the stack one still says EQL 2.3.0-pre.3.Checks
generate-docs:eql+generate-docs:eql-apiagainst the real 3.0.3 release manifest — strict drift check greenbunx next build— cleanvalidate-content,validate-links,validate-mermaid,types:check— cleanbiome checkon every changed file — clean (the 6 repo-wide lint errors are pre-existing, in files this PR doesn't touch)