Skip to content

docs: EQL 3.0.4 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface - #74

Merged
coderdan merged 6 commits into
v2from
docs/eql-grouped-value
Jul 28, 2026
Merged

docs: EQL 3.0.4 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface#74
coderdan merged 6 commits into
v2from
docs/eql-grouped-value

Conversation

@coderdan

@coderdan coderdan commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Started as encrypt-query-language#423 (eql_v3.grouped_value), and grew into the EQL 3.0.3 pin bump once 3.0.3 shipped. Three related things, all driven by the same root cause: the docs were pinned to 3.0.0, and 3.0.1–3.0.3 changed surfaces the hand-written pages teach.

1. eql_v3.grouped_value and the GROUP BY / DISTINCT shapes

grouped_value projects an encrypted column while grouping by its equality term. Writing it up surfaced two wrong claims: that GROUP BY email and SELECT DISTINCT email work in natural form, and that grouping compares equality terms. Neither holds — an encrypted column is a domain over jsonb with no operator class of its own, so Postgres compares randomised ciphertext and every row lands in its own group, silently.

  • Group on eql_v3.eq_term(col) for correctness first, planner economics second. The old "raise work_mem if an ORM forces the raw form" advice is gone — it tuned a query that was already wrong.
  • eql_v3.grouped_value: the 42803 rejection it fixes, the shape that fixes it, and its semantics (first non-NULL per group, all-NULLNULL, PARALLEL SAFE, input returned unchanged so it decrypts client-side).
  • Dedup is DISTINCT ON (eql_v3.eq_term(col)) col with a matching leading ORDER BY; plain DISTINCT isn't supported, and DISTINCT ON has no projection restriction, so grouped_value isn't involved there.

2. Text fuzzy match: @@ / eql_v3.matches (U-008, U-011)

3.0.1 replaced @> / <@ and eql_v3.contains / contained_by on text_match / text_search / text_search_ore with the directional @@ / eql_v3.matches; the old spellings now raise. The docs still taught them — on text.mdx, filtering.mdx, core-concepts.mdx and indexes.mdx.

3.0.3 then added the U-011 guard, documented here as a LIKE truth table: a search term below the tokeniser floor (no trigrams, bf: []) used to match every row in the table, and now matches only empty-bloom values. The one regression it carries is documented too — matches is non-STRICT and names the needle twice, so a needle from an uncorrelated subquery stops inlining and loses the GIN index; bind parameters and literals are unaffected.

3. The encrypted-JSON surface (U-007, U-010)

Not in the original scope, but the pin bump made it a build gate — and these pages have been wrong for every reader on a released 3.0.x:

  • eql_v3_jsoneql_v3_json_search, eql_v3_jsonb_entryeql_v3_json_entry, query_jsonbquery_json. The bare eql_v3_json is now a storage-only domain.
  • Field equality is value-selector containment, not = on an extracted leaf, which raises. eq_term on an entry is a deprecated alias for ope_term returning OPE bytes — so extracted JSON leaves can no longer be grouped by equality at all, which the grouping page now states outright instead of showing an example that doesn't work.
  • Wire format: document-level key header h, no root c, selector-derived nonces, sentinel value entries, hm retired — a re-encryption, not a rename.
  • jsonb_array_elements_text was removed; entry ciphertext isn't independently decryptable.

Tooling

  • A silent hole in the generated catalog: the fragment generator keyed the match card off a match capability the 3.0.1 catalog no longer emits (text_match reports capabilities: ["storage"] with supportedOperators: ["@@"]), so the text function reference had quietly lost its match entry. It now keys off the operator and renders eql_v3.matches.
  • Drift check exempts lines that name a symbol in order to say it's gone, mirroring validate-content-api.ts, so a rename note can name the old symbol.
  • Two manifest blind spots, both verified in the 3.0.3 install SQL: eql_v3.grouped_value (hand-written CREATE AGGREGATE — the catalog only contributes the 60 generated min/max aggregates, so 61 ship and 60 are listed) and public.eql_v3_json (created in a DO block, same cause as the query-operand domains). Both look like upstream manifest gaps worth fixing in the EQL repo — the generated API reference is missing them entirely.
  • BadExample component (src/components/bad-example.tsx), rustdoc compile_fail-style: red frame, an explicit label distinguishing "Raises" from "Silently wrong", and the verbatim error. Used for the four broken examples on these pages.

Generated artifacts (functions.mdx, the partials, eql-version.ts, the stack EQL API page) are deliberately not committed: prebuild regenerates them from the pinned release, and the committed copies are placeholders — the stack one still says EQL 2.3.0-pre.3.

Checks

  • generate-docs:eql + generate-docs:eql-api against the real 3.0.3 release manifest — strict drift check green
  • bunx next build — clean
  • validate-content, validate-links, validate-mermaid, types:check — clean
  • biome check on every changed file — clean (the 6 repo-wide lint errors are pre-existing, in files this PR doesn't touch)

…hapes

EQL #423 added `eql_v3.grouped_value`, the aggregate that lets a query project
an encrypted column while grouping by that column's equality term. Documenting
it exposed two wrong claims on the grouping page: that `GROUP BY email` and
`SELECT DISTINCT email` work in natural form, and that grouping compares
equality terms. Neither is true — an encrypted column is a domain over jsonb
with no custom operator class, so Postgres compares the raw (randomised)
ciphertext and never collapses two encryptions of the same value.

- Group on `eql_v3.eq_term(col)` for correctness first, planner economics
  second; drop the "raise work_mem if an ORM forces the raw form" advice, which
  rescued the performance of a query that was already returning wrong answers.
- New `grouped_value` section: the 42803 rejection it fixes, the shape that
  fixes it, and its semantics (first non-null per group, all-NULL group is
  NULL, parallel safe, returns its input unchanged).
- Deduplication is `DISTINCT ON (eql_v3.eq_term(col)) col` with the matching
  leading ORDER BY. Plain DISTINCT is not supported; `DISTINCT ON` has no
  projection restriction, so `grouped_value` is not involved there.
- Same correction to the Indexes page's GROUP BY section, and grouped_value in
  the MIN/MAX and JSON-leaf examples.

grouped_value landed after eql-3.0.2, so the page carries a version callout
(with the 3.0.x join-back workaround) and the drift check gets an UNRELEASED
allowlist — reported on every run, and deleted when EQL_RELEASE_TAG is bumped
to a release whose manifest covers it.
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
public-docs Ready Ready Preview, Comment Jul 28, 2026 7:36am

Request Review

Comment thread content/docs/reference/eql/grouping-and-aggregates.mdx Outdated
Comment thread content/docs/reference/eql/grouping-and-aggregates.mdx
Comment thread content/docs/reference/eql/grouping-and-aggregates.mdx Outdated
…rsion note

- Restore the `GROUP BY` and `DISTINCT` heading, and fold deduplication back
  under it: one section for how both operators must be written, with the
  `grouped_value` projection following as its own section.
- Add a `BadExample` component (rustdoc `compile_fail` styling: red frame,
  explicit label, verbatim error) so a broken example can't be mistaken for a
  working one while skimming, and use it for the two broken shapes here — the
  natural `GROUP BY` / `DISTINCT` forms, and the bare column under GROUP BY.
- Drop the unreleased-version callout.
Documenting U-011 (the empty-bloom fuzzy-match guard, shipped in 3.0.3) meant
moving EQL_RELEASE_TAG off 3.0.0, and the strict drift check then reported what
the pin had been hiding: 3.0.1 renamed the whole encrypted-JSON surface and
replaced the text match operator, and the hand-written pages still taught the
3.0.0 spelling of both. Every customer on a released 3.0.x has been reading
docs whose text-match and JSON examples raise.

Text fuzzy match (U-008, U-011):
- `@>` / `<@` and `eql_v3.contains` / `contained_by` on text_match,
  text_search and text_search_ore are now `@@` / `eql_v3.matches`; the old
  spellings raise, and the pages say so.
- Document the empty-needle guard as a LIKE truth table: a sub-floor search
  term (no trigrams, `bf: []`) used to match every row and now matches only
  empty-bloom values.
- Note the one regression it carries: `matches` is non-STRICT and names the
  needle twice, so a needle from an uncorrelated subquery stops inlining and
  loses the GIN index. Bind parameters and literals are unaffected.

Encrypted JSON (U-007, U-010):
- eql_v3_json → eql_v3_json_search, eql_v3_jsonb_entry → eql_v3_json_entry,
  query_jsonb → query_json; the bare eql_v3_json is now storage-only.
- Field equality is value-selector containment, not `=` on an extracted leaf
  (which raises). `eq_term` on an entry is a deprecated alias for `ope_term`
  and is not an exact equality term — so JSON leaves can no longer be grouped
  by equality at all, which the grouping page now states outright.
- Wire format: document-level key header `h`, no root `c`, selector-derived
  nonces, sentinel value entries, `hm` retired.
- `jsonb_array_elements_text` was removed; entry ciphertext is not
  independently decryptable.

Tooling:
- The fragment generator keyed the match card off a `match` capability the
  3.0.1 catalog no longer emits (text_match now reports `["storage"]` with
  `supportedOperators: ["@@"]`), so the text function catalog silently lost its
  match entry. It now keys off the operator and renders `eql_v3.matches`.
- Drift check: exempt lines that name a symbol in order to say it is gone, so
  a rename note can name the old symbol — the same line-local exemption
  validate-content-api.ts makes.
- Two more manifest blind spots, both verified present in the 3.0.3 install
  SQL: `eql_v3.grouped_value` (hand-written CREATE AGGREGATE; the catalog only
  contributes the 60 generated min/max aggregates) and `public.eql_v3_json`
  (created in a DO block, like the query-operand domains).

Generated artifacts (functions.mdx, the partials, eql-version.ts, the stack
EQL API page) are left alone: prebuild regenerates them from the pinned
release, and the committed copies are placeholders — the stack one still says
EQL 2.3.0-pre.3.
@coderdan coderdan changed the title docs: document eql_v3.grouped_value and fix the GROUP BY / DISTINCT shapes docs: EQL 3.0.3 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface Jul 27, 2026
EQL 3.0.3's ope_term brief contains "= / <> are blocked". Doxygen comments
are written for a plain-text renderer, so bare SQL operators are normal in
them — but the generator dropped them into functions.mdx verbatim, where MDX
read `<>` as an unclosed JSX fragment and failed the build on the generated
content file rather than at the source.

Escape `<` and `{` in every prose string the manifest supplies (brief,
description, param and return descriptions), skipping inline-code spans so
the descriptions that already write `<>` in backticks keep rendering as code.
coderdan added 2 commits July 28, 2026 17:31
EQL 3.0.4 ships the manifest-extraction fixes from
cipherstash/encrypt-query-language#427, so `eql_v3.grouped_value` and
`public.eql_v3_json` now resolve from the catalog and no longer need drift-check
exemptions. Verified against the released manifest: grouped_value(jsonb) and
lints() present, 53 domains (was 52) including public.eql_v3_json, and the
`match` capability restored to text_match and both text_search variants.

Only `eql_v3.query_text_eq` stays on the allowlist — the 40 scalar
query-operand domains created in a `DO ... EXECUTE` block are a separate gap
that 3.0.4 does not address. Removing the other two means a regression in
either now fails the drift check instead of staying quiet.

Also always stamp the pinned tag as the manifest's version rather than only
substituting when it reads "DEV". EQL's release workflow derives that field
from a tag input that can arrive empty, in which case json.sh falls back to
"DEV" — which 3.0.4 shipped, and which would otherwise render as "EQL DEV" in
the banner on every reference page. The existing guard already covered that
case; this extends it to warn when a manifest is stamped with a DIFFERENT
release, which would be a packaging mix-up worth seeing rather than a cosmetic
default worth silencing.

bun run build passes: drift check clean against 3.0.4, 1990 functions,
banner reads EQL 3.0.4.

Claude-Session: https://claude.ai/code/session_01NkuQNMvw9BpB4BWWeKtfV8
@coderdan coderdan changed the title docs: EQL 3.0.3 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface docs: EQL 3.0.4 — grouped_value, @@ fuzzy match, and the encrypted-JSON surface Jul 28, 2026
@coderdan

Copy link
Copy Markdown
Contributor Author

Rebased on current v2 and repinned from 3.0.3 to 3.0.4 (90916c4).

eql-3.0.4 ships the manifest-extraction fixes from cipherstash/encrypt-query-language#427, so the two drift-check exemptions this PR previously needed are gone:

-const MANIFEST_BLIND_SPOTS = new Set([
-  "eql_v3.query_text_eq",
-  "eql_v3.grouped_value",
-  "public.eql_v3_json",
-]);
+const MANIFEST_BLIND_SPOTS = new Set(["eql_v3.query_text_eq"]);

Verified against the released manifest: grouped_value(jsonb) and lints() present, 53 domains (was 52) including public.eql_v3_json, and the match capability restored to text_match and both text_search variants. eql_v3.query_text_eq stays — the 40 DO-block query-operand domains are a separate gap 3.0.4 doesn't touch. Both removals mean a regression now fails the drift check instead of being silently tolerated.

One extra change: the pinned tag is now always stamped as the manifest's version, not only when it reads DEV. 3.0.4 shipped with "version": "DEV" (3.0.3 was correct), because EQL's release workflow derives that field from a tag input that can arrive empty and json.sh falls back to DEV silently — filed as cipherstash/encrypt-query-language#430. The existing guard already covered DEV; this extends it to warn when a manifest is stamped with a different release, which would be a packaging mix-up worth seeing rather than a default worth silencing.

bun run build passes end to end: drift check clean against 3.0.4, 1990 functions, banner reads EQL 3.0.4, all validators green, Vercel green.

@coderdan
coderdan merged commit 13da74e into v2 Jul 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant