diff --git a/README.md b/README.md index f5b514aaf..bc9b33043 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,9 @@ Store encrypted data alongside your existing data: - [Local development (fastest)](#local-development-fastest) - [Install into an existing database](#install-into-an-existing-database) - [dbdev](#dbdev) +- [EQL Components](#eql-components) + - [Release artifacts](#release-artifacts) +- [Database Permissions](#database-permissions) - [Getting started](#getting-started) - [Enable encrypted columns](#enable-encrypted-columns) - [Encrypt configuration](#encrypt-configuration) @@ -72,7 +75,8 @@ EQL installs the following components into the `eql_v3` schema: | --------------------------------------------------- | ------------- | ------------------------------------------------------------------- | | `eql_v3` | Schema | Holds EQL operators, term extractors, comparison wrappers, and aggregates | | `public.`, `public._eq`, `public._ord` | Domain types | Per-scalar encrypted columns (one family per scalar: `integer`, `text`, `timestamp`, …) | -| `public.eql_v3_json` | Domain type | Encrypted JSON (structured-encryption) documents | +| `public.eql_v3_json_search` | Domain type | Searchable encrypted JSON (structured-encryption) documents | +| `public.eql_v3_json` | Domain type | Encrypted JSON, storage-only (no query surface) | | `eql_v3.eq_term` / `ord_term` / `match_term` | Functions | Index-term extractors for functional indexes | @@ -80,14 +84,14 @@ EQL installs the following components into the `eql_v3` schema: The `eql_v3` schema holds the operators, term extractors, comparison wrappers, and `MIN` / `MAX` aggregates for the encrypted-domain types. The encrypted-domain types themselves live in `public` (see below), and the internal SEM index-term types live in `eql_v3_internal`. -Encrypted columns are typed as `public` domains (e.g. `public.eql_v3_text_eq`, `public.eql_v3_json`), and the searchable surface available on a column is fixed by its domain **variant** — there is no database-side configuration state. Which index terms a value carries is decided by the encryption client (CipherStash Stack / CipherStash Proxy). +Encrypted columns are typed as `public` domains (e.g. `public.eql_v3_text_eq`, `public.eql_v3_json_search`), and the searchable surface available on a column is fixed by its domain **variant** — there is no database-side configuration state. Storage-only variants (`public.eql_v3_text`, `public.eql_v3_json`, …) carry no query surface at all; the searchable JSON domain is `public.eql_v3_json_search`. Which index terms a value carries is decided by the encryption client (CipherStash Stack / CipherStash Proxy). -The domain types deliberately live in `public`, not `eql_v3`, so application tables survive an EQL uninstall: `DROP SCHEMA eql_v3 CASCADE` removes the operators, extractors, and aggregates but leaves the `public`-typed columns (and their data) intact. Re-running the install script is idempotent. +The domain types deliberately live in `public`, not `eql_v3`, so application tables survive an EQL uninstall: `DROP SCHEMA eql_v3 CASCADE` removes the operators, extractors, and aggregates but leaves the `public`-typed columns (and their data) intact. Re-running the install script is safe for columns and data — but note it begins with that same `DROP SCHEMA eql_v3 CASCADE`, which also **cascade-drops any functional indexes** built on the `eql_v3` extractors. After a re-install or upgrade, re-create your encrypted-column indexes and `ANALYZE` (see [Database Indexes](docs/reference/database-indexes.md)). ### Release artifacts -EQL v3 prereleases can ship three artifacts under one identity: the SQL + docs +EQL v3 releases ship three artifacts under one identity: the SQL + docs GitHub release, the Rust `eql-bindings` crate, and the TypeScript `@cipherstash/eql` npm package. The language packages bundle the exact SQL installer they were generated against. @@ -95,69 +99,23 @@ installer they were generated against. ## Database Permissions -EQL requires specific database privileges to install and operate correctly. The permissions needed depend on your deployment pattern. +The canonical reference is **[EQL permissions & grants](docs/reference/permissions.md)** — install privileges (including the one superuser-gated component, the ORE operator class, and which managed platforms allow it), the per-query-path grant matrix, and the operator-free function equivalents. The short version: -### Default Permissions (Recommended) - -For most use cases, grant the following permissions to the database user that will install and use EQL: +- **Installing** needs `CREATE` on the database plus `CREATE`/`USAGE` on `public` — no superuser, except for the optional block-ORE operator class. Altering the tables that receive encrypted columns requires table *ownership* (PostgreSQL has no grantable `ALTER` table privilege). +- **The installer issues no `GRANT`s.** Access to `eql_v3` / `eql_v3_internal` is strictly opt-in per role. +- **A runtime role** that queries encrypted columns typically needs: ```sql --- Database-level permissions -GRANT CREATE ON DATABASE your_database TO your_eql_user; - --- Schema permissions -GRANT USAGE ON SCHEMA public TO your_eql_user; -GRANT CREATE ON SCHEMA public TO your_eql_user; - --- User table permissions (for encrypted column constraints) -GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_eql_user; --- Or grant ALTER on specific tables that will have encrypted columns: --- GRANT ALTER ON TABLE your_table TO your_eql_user; -``` - -**Why these permissions are needed:** - -- **CREATE ON DATABASE**: Required to create the `eql_v3` schema, domain types, and functions during installation -- **CREATE ON SCHEMA public**: Required to add encrypted columns (typed as `public` domains) to tables in the public schema -- **ALTER on user tables**: encrypted-domain `CHECK` constraints are validated on the user tables - -### Splitting Read and Write Access - -A common production pattern separates setup/migration permissions from runtime permissions: - -#### Setup/Migration User (Write Access) - -Use during database migrations and EQL installation: - -```sql --- All default permissions above, plus: -GRANT CREATE ON DATABASE your_database TO your_migration_user; -GRANT CREATE ON SCHEMA public TO your_migration_user; -GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_migration_user; -``` - -#### Runtime User (Read Access) - -Use for application queries in production: - -```sql --- EQL schema usage (resolves the encrypted operators / extractors) GRANT USAGE ON SCHEMA eql_v3 TO your_app_user; GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3 TO your_app_user; - --- eql_v3_internal holds the implementation the public eql_v3 operators dispatch --- into. Grant it the same way as eql_v3 — explicitly, per runtime role. +-- The public operators inline into eql_v3_internal, so query roles need it too: GRANT USAGE ON SCHEMA eql_v3_internal TO your_app_user; GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3_internal TO your_app_user; - -- User table access (normal application permissions) GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE your_tables TO your_app_user; ``` -**Migration Workflow:** -1. Use the migration user to install EQL and add encrypted columns -2. Use the runtime user for normal application operations -3. Schema changes (adding/removing encrypted columns) require the migration user +See [permissions.md](docs/reference/permissions.md) for the pgcrypto-schema grant the ORE variants additionally need, and for narrowing grants per query path. ### dbdev @@ -173,7 +131,7 @@ Once EQL is installed in your PostgreSQL database, you can start using encrypted ### Enable encrypted columns -Define encrypted columns using a `public` encrypted-domain type. Type the column as the **variant** for the capability you need — `public.eql_v3_text_eq` for equality, `public._ord` for range/ordering, `public.eql_v3_text_match` for full-text, `public.eql_v3_json` for encrypted JSON. Each is stored as `jsonb` with a `CHECK` constraint that validates the encrypted payload. +Define encrypted columns using a `public` encrypted-domain type. Type the column as the **variant** for the capability you need — `public.eql_v3_text_eq` for equality, `public._ord` for range/ordering, `public.eql_v3_text_match` for full-text, `public.eql_v3_json_search` for searchable encrypted JSON (`public.eql_v3_json` is the storage-only flavour). Each is stored as `jsonb` with a `CHECK` constraint that validates the encrypted payload. **Example:** @@ -202,7 +160,7 @@ In order to enable searchable encryption, you will need to configure your Cipher ## Performance -Query latency for searchable-encryption operations stays low across data set sizes. The numbers below are query-only medians (no decryption) from a full benchmark run against EQL 2.3 on PostgreSQL 17, across four row-count tiers. +Query latency for searchable-encryption operations stays low across data set sizes. The numbers below are query-only medians (no decryption) from a full benchmark run against EQL 2.3 on PostgreSQL 17, across four row-count tiers. (The JSON rows predate the 3.0 ste_vec redesign — field equality is now value-selector containment — so treat them as indicative, not current.) | Family | Scenario | 10k | 100k | 1M | 10M | |---|---|--:|--:|--:|--:| @@ -314,7 +272,7 @@ To upgrade to the latest version of EQL, you can simply run the install script a ``` > [!NOTE] -> The install script will not remove any existing configurations, so you can safely run it multiple times. +> Re-running the install script is safe for your columns and data (the domain types live in `public` and survive). It does, however, begin with `DROP SCHEMA eql_v3 CASCADE`, which cascade-drops any **functional indexes** built on the `eql_v3` extractors — re-create them and `ANALYZE` after every upgrade (see [Database Indexes](docs/reference/database-indexes.md)). #### Using dbdev? @@ -326,11 +284,11 @@ Follow the instructions in the [dbdev documentation](https://database.dev/cipher **A query returns no rows / silently runs native `jsonb` semantics** - **Cause**: the query operand was an untyped literal, so PostgreSQL flattened the `eql_v3` domain to its `jsonb` base type and resolved the native operator -- **Solution**: type the operand — `WHERE col = $1::public.eql_v3_text_eq` (CipherStash Proxy supplies typed parameters automatically) +- **Solution**: type the operand with the matching query-operand domain — `WHERE col = $1::eql_v3.query_text_eq` (CipherStash Proxy supplies typed parameters automatically) **Error: "operator not supported" (raised)** - **Cause**: the operator is blocked for the column's domain variant (e.g. `<` on an `_eq` column, or `LIKE` on any encrypted column) -- **Solution**: type the column as a variant that carries the required term (see the [SQL support matrix](docs/reference/sql-support.md)); use `@>` rather than `LIKE` for text match +- **Solution**: type the column as a variant that carries the required term (see the [SQL support matrix](docs/reference/sql-support.md)); use `@@` rather than `LIKE` for text match **`=` returns no rows on a populated column** - **Cause**: the column's values do not carry an `hm` equality term diff --git a/crates/eql-bindings/README.md b/crates/eql-bindings/README.md index f7c9536ac..4812c331e 100644 --- a/crates/eql-bindings/README.md +++ b/crates/eql-bindings/README.md @@ -33,8 +33,9 @@ Shared wire fields are reusable newtypes in | Newtype | Wire key | Inner | Backs | |---------|----------|-------|-------| | `Ciphertext` | `c` | `String` | every domain (envelope) | -| `Hmac256` | `hm` | `String` | `_eq` domains | -| `OreBlock256` | `ob` | `Vec` | `_ord` / `_ord_ore` domains | +| `Hmac256` | `hm` | `String` | `_eq` domains + the text ordering/search domains | +| `OpeCllw` | `op` | `String` | `_ord` / `_ord_ope` domains, `text_search` | +| `OreBlock256` | `ob` | `Vec` | `_ord_ore` domains, `text_search_ore` | | `BloomFilter` | `bf` | `Vec` (signed!) | `_match` domains | Note "v3" names the SQL schema generation (`eql_v3.*`); the JSON envelope diff --git a/docs/README.md b/docs/README.md index 7dccd6340..8b7a6c1d9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,12 +9,12 @@ This directory contains the documentation for the Encrypt Query Language (EQL). ## Reference - [EQL Functions Reference](reference/eql-functions.md) - Complete API reference for all EQL functions -- [SQL support matrix](reference/sql-support.md) - Which SQL operators and features each encrypted index enables -- [Database Indexes for Encrypted Columns](reference/database-indexes.md) - PostgreSQL B-tree index creation and usage +- [SQL support matrix](reference/sql-support.md) - Which SQL operators and features each encrypted-domain variant enables +- [Database Indexes for Encrypted Columns](reference/database-indexes.md) - Index recipes (hash / btree / GIN), engagement rules, and build performance for encrypted columns - [Writing fast queries against EQL columns](reference/query-performance.md) - Performance overview (points to Database Indexes) - [Adding a Scalar Encrypted-Domain Type](reference/adding-a-scalar-encrypted-domain-type.md) - How the `eql_v3.` domain families are generated - [EQL with JSON and JSONB](reference/json-support.md) -- [EQL payload / wire format](../crates/eql-bindings/README.md) - Canonical wire types for the encrypted payload (envelope `v`/`i`/`c` and the `hm`/`ob`/`bf` index terms) +- [EQL payload / wire format](../crates/eql-bindings/README.md) - Canonical wire types for the encrypted payload (envelope `v`/`i`/`c` and the `hm`/`op`/`ob`/`bf` index terms) - [Client-side index configuration](https://cipherstash.com/docs/stack/cipherstash/encryption/schema) - Configuring searchable encryption in CipherStash Stack / CipherStash Proxy ## Tutorials diff --git a/docs/concepts/WHY.md b/docs/concepts/WHY.md index f27311e9e..93cfcfdd3 100644 --- a/docs/concepts/WHY.md +++ b/docs/concepts/WHY.md @@ -22,7 +22,7 @@ EQL enables encryption in use without significant changes to your application co A variety of searchable encryption techniques are available, including: - **Matching** - Equality or partial matches -- **Ordering** - comparison operations using order revealing encryption +- **Ordering** - comparison operations using order-preserving encryption (CLLW-OPE by default; block-ORE as an opt-in variant) - **Uniqueness** - enforcing unique constraints - **Containment** - containment queries using structured encryption diff --git a/docs/development/sql-documentation.md b/docs/development/sql-documentation.md index 9e1c1729a..6f7536dc5 100644 --- a/docs/development/sql-documentation.md +++ b/docs/development/sql-documentation.md @@ -35,14 +35,14 @@ prefix (not plain `--`). Coverage and required tags are checked by `mise run doc --! bare-form queries. --! --! @param a public.eql_v3_integer_eq Encrypted value carrying an `hm` term ---! @return eql_v3.hmac_256 The equality index term +--! @return eql_v3_internal.hmac_256 The equality index term --! --! @example --! CREATE INDEX ON users USING hash (eql_v3.eq_term(salary_eq)); --! --! @see eql_v3.ord_term CREATE FUNCTION eql_v3.eq_term(a public.eql_v3_integer_eq) - RETURNS eql_v3.hmac_256 + RETURNS eql_v3_internal.hmac_256 AS $$ ... $$; ``` @@ -88,9 +88,9 @@ CREATE OPERATOR = ( ```sql --! @brief Encrypted-domain type for an equality-searchable integer column --! ---! A `jsonb`-backed domain in the `eql_v3` schema. The `CHECK` requires the +--! A `jsonb`-backed domain in the `public` schema. The `CHECK` requires the --! envelope keys (`v`, `i`, `c`), the equality term (`hm`), and pins the ---! payload version (`VALUE->>'v' = '2'`). +--! payload version (`VALUE->>'v' = '3'`). --! --! @see eql_v3.eq_term CREATE DOMAIN public.eql_v3_integer_eq AS jsonb @@ -104,7 +104,7 @@ CREATE DOMAIN public.eql_v3_integer_eq AS jsonb --! @param $1 public.eql_v3_integer_ord Accumulated state --! @param $2 public.eql_v3_integer_ord New value --! @return public.eql_v3_integer_ord Updated state -CREATE FUNCTION eql_v3.min_sfunc(public.eql_v3_integer_ord, public.eql_v3_integer_ord) +CREATE FUNCTION eql_v3_internal.min_sfunc(public.eql_v3_integer_ord, public.eql_v3_integer_ord) RETURNS public.eql_v3_integer_ord AS $$ ... $$; @@ -119,10 +119,12 @@ AS $$ ... $$; --! @example --! SELECT eql_v3.min(price_encrypted) FROM products; --! ---! @see eql_v3.min_sfunc +--! @see eql_v3_internal.min_sfunc CREATE AGGREGATE eql_v3.min(public.eql_v3_integer_ord) ( - SFUNC = eql_v3.min_sfunc, - STYPE = public.eql_v3_integer_ord + SFUNC = eql_v3_internal.min_sfunc, + STYPE = public.eql_v3_integer_ord, + COMBINEFUNC = eql_v3_internal.min_sfunc, + PARALLEL = safe ); ``` @@ -186,8 +188,11 @@ CREATE FUNCTION eql_v3."[operator]"(...) --! --! @see eql_v3.eq_term --! @note This is a transient type used only during query execution ---! @note Encrypted-domain types are jsonb-backed — always `AS jsonb`, never domain-over-domain -CREATE DOMAIN eql_v3.[type_name] AS jsonb; +--! @note SEM index-term types live in `eql_v3_internal` and are backed by whatever +--! base type the term needs (`hmac_256 AS text`, `ope_cllw AS bytea`) — never +--! domain-over-domain. (Encrypted-domain *column* types are the jsonb-backed +--! `public.eql_v3_*` domains.) +CREATE DOMAIN eql_v3_internal.[type_name] AS [base_type]; ``` ### Template: Aggregate Function diff --git a/docs/reference/adding-a-scalar-encrypted-domain-type.md b/docs/reference/adding-a-scalar-encrypted-domain-type.md index dfc5952a9..c04475906 100644 --- a/docs/reference/adding-a-scalar-encrypted-domain-type.md +++ b/docs/reference/adding-a-scalar-encrypted-domain-type.md @@ -1,7 +1,8 @@ # Adding a Scalar Encrypted-Domain Type The one reference for adding a scalar encrypted-domain type (`integer`, `smallint`, -and future ordered numeric scalars). The **top half** (§§1–4) is the path you +`bigint`, `date`, `timestamp`, `numeric`, `real`, `double`, and future +scalars). The **top half** (§§1–4) is the path you follow to add a type; the **reference half** (§§5–8) is the detail behind it — the generated surface, its invariants, and how the generator itself works. Read top-down to ship a type; drop into the reference half when something @@ -37,7 +38,8 @@ storage-only scalar (see §7). ## 1. TL;DR — the one path -To add a scalar type `` (e.g. `bigint`), with Rust type `` (e.g. `i64`): +To add a scalar type `` (e.g. a hypothetical `uuid`), with Rust type `` +(e.g. `uuid::Uuid`): 1. **Add a `DomainFamily` row to `eql_domains::CATALOG`** — just `name` + `domains` — plus a matching `TypeFixtures` record (carrying the `kind` and the @@ -132,8 +134,24 @@ by the type system and the catalog `#[test]`s rather than a runtime validator: Pinned by `every_domain_name_starts_with_its_family_name`. - **`domains`** (on `DomainFamily`) — a non-empty `&[Domain]` (pinned by `every_type_has_at_least_one_domain`), each a bare `name` + the fixed `&[Term]` it - carries. The storage domain is `name: ""` with no terms; `eq => [Term::Hm]`; - `ord` and `ord_ope => [Term::Ope]`; `ord_ore => [Term::Ore]`. A `Domain` declares nothing else — no + carries + a `shape` (always `Shape::Scalar` for a scalar family; + `Shape::SteVec` exists only for the hand-written `json` document domains). + The storage domain is `name: ""` with no terms. The **ordered-numeric** shape + (`ORDERED_INT_DOMAINS`) is: `eq => [Term::Hm]`; `ord` and + `ord_ope => [Term::Ope]`; `ord_ore => [Term::Ore]`. That + ordering-term-only rule is admissible **iff ordering-term equality is + lossless for the kind** — the term is deterministic *and* injective, so term + equality exactly mirrors plaintext equality. That holds for every + numeric/date/timestamp kind, which is why their generated `=` may inline to + `ord_term(a) = ord_term(b)` (`integer_ord_functions.sql`). For a + **non-lossless** kind — `text`, and any future string-like scalar — every + eq-capable domain must **lead with `Hm`** (`TEXT_DOMAINS`: `ord` and + `ord_ope => [Term::Hm, Term::Ope]`; `ord_ore => [Term::Hm, Term::Ore]`), so + `=`/`<>` resolve through `eq_term` (exact HMAC, `text_ord_functions.sql`) and + never the ordering term — otherwise the generated `=` silently returns wrong + answers for plaintexts whose ordering terms don't faithfully mirror + equality. A contributor adding a new string-like scalar must follow the + `text` shape. A `Domain` declares nothing else — no extractor names, no operator lists, no REQUIRE edges. Every behavioural fact comes from the `Term` enum. - **`kind`** (on the `TypeFixtures` record) — a `ScalarKind` (`I16` / `I32` / @@ -153,7 +171,9 @@ by the type system and the catalog `#[test]`s rather than a runtime validator: - **`values`** (on the `TypeFixtures` record) — the type's plaintext fixture list (see below). -**Terms** are fixed by the `Term` enum (`crates/eql-domains/src/lib.rs`). The +**Terms** are fixed by the `Term` enum (declared in +`crates/eql-domains/src/lib.rs`; its `impl` methods live in +`crates/eql-domains/src/term.rs`). The `json_key` / `extractor` / `ctor` values are the cross-schema SQL contract (the Returns column below is `eql_v3_internal.` + `ctor` — SEM index-term types live in the internal schema) — changing one is a generated-SQL @@ -175,7 +195,8 @@ one extractor.) A type that needs a non-ORE equality term on an ordered domain needs a **new `Term`**, not a catalog flag. Adding a term is a code change to the `Term` -enum's `impl` methods (`json_key`, `extractor`, `ctor`, `role`, +enum's `impl` methods in `crates/eql-domains/src/term.rs` (`json_key`, +`extractor`, `ctor`, `binding_newtype`, `role`, `operators`, `requires`) with matching `#[test]`s (`term_tests` / `term_helper_tests`) — never a free-form catalog field. @@ -337,7 +358,7 @@ integer kinds: | File | Add | |------|-----| -| `tests/sqlx/src/scalar_types.rs` | One ` => ` line in the `scalar_types!` list (e.g. `bigint => i64,`). This single line drives the `impl ScalarType` **(integer kinds only)**, the `eql_v3_` fixture module, the `scalar_matrix!` suite, and the `generate_for_token` arm — all generated by the `eql-tests-macros` proc-macros. | +| `tests/sqlx/src/scalar_types.rs` | One ` => ` line in the `scalar_types!` list (e.g. `uuid => uuid::Uuid,`). This single line drives the `impl ScalarType` **(integer kinds only)**, the `eql_v3_` fixture module, the `scalar_matrix!` suite, and the `generate_for_token` arm — all generated by the `eql-tests-macros` proc-macros. | | `tests/sqlx/src/fixtures/eql_plaintext.rs` | A sealed `EqlPlaintext` impl for ``: `impl Sealed for {}` and `impl EqlPlaintext for ` carrying just `const KIND: ScalarKind` plus the value-typed `to_plaintext` → the right `Plaintext` variant. `CAST` and `PLAINTEXT_SQL_TYPE` are **derived** from `KIND` via the `cast_for_kind` / `plaintext_sql_type_for_kind` `const fn` defaults, so a brand-new kind needs an arm in those two helpers — not a per-type const (see §3.1 for a non-integer kind's full wiring). Keep the three `#[test]`s (cast / sql-type / to_plaintext) mirroring the existing ones. | | `tests/sqlx/src/scalar_domains.rs` **(non-integer only)** | The `impl ScalarType` the proc-macro skips for non-integer kinds. For a **chrono-backed** kind (`date`, `timestamp`) this is a `temporal_values!` invocation that materialises the catalog ISO/RFC3339 strings into a `LazyLock>` and emits `impl ScalarType` + `OrderedScalar` (+ `SignedScalar` for `date` and `timestamp`). For **`text`** it is a hand-written `impl ScalarType` / `OrderedScalar` block (an overridden lexicographic-median `mid_pivot()` — `min`/`max` inherit the fixture-derived defaults — plus a `to_sql_literal` override) — `String` has no numeric origin, so it is deliberately **not** `SignedScalar`. | @@ -616,7 +637,10 @@ Whether an operator routes to a wrapper or a blocker is a per-domain decision driven by the domain's terms (`Term::operators_for_terms`), not a property of the operator. Supported operators are emitted with full planner metadata (`COMMUTATOR`, `NEGATOR`, `RESTRICT`, `JOIN` selectivity estimators) backing -onto inlinable wrappers; everything else carries minimal metadata backing onto +onto inlinable wrappers — except the directional `@@` fuzzy match, whose +`match_metadata()` deliberately carries no commutator or negator (there is no +reverse operator), only the containment selectivity estimators; everything +else carries minimal metadata backing onto blockers. Path operators always back onto blockers — neither current term enables them — and the native `jsonb` operators are blocker-only **except the three symmetric `@@` overloads**, which back onto the inlinable bloom fuzzy-match @@ -791,11 +815,13 @@ adding a type. ### Why a generator -A single scalar type emits several hundred SQL declarations. For `integer`: fourteen -files, five domains, four extractor functions, dozens of wrappers and blockers, 220 -`CREATE OPERATOR` statements (44 per domain), and MIN/MAX aggregates per ordered -domain. (The per-domain figure is fixed — 44 `CREATE OPERATOR` statements per domain, the `1 + 2D + -A` file formula below — so a type with more domains, e.g. `text`'s seven, scales +A single scalar type emits several hundred SQL declarations. For `integer`: twenty-five +files, five column domains plus four `eql_v3.query_*` operand domains, four +extractor functions, dozens of wrappers and blockers, 235 +`CREATE OPERATOR` statements across the column domains (47 per domain), the +query-operand and `json_entry` cross surfaces, and MIN/MAX aggregates per ordered +domain. (The per-domain figure is fixed — 47 `CREATE OPERATOR` statements per +column domain, the file formula below — so a type with more domains, e.g. `text`'s eight, scales those totals up.) The shape is mechanical and the invariants are unforgiving — a `STRICT` blocker silently bypasses its exception; a pinned `search_path` reverts queries to seq @@ -830,16 +856,21 @@ columns survive an `eql_v3` uninstall. `SCHEMA = "eql_v3"` the extractors, comparison wrappers, and aggregates — while `INTERNAL_SCHEMA = "eql_v3_internal"` qualifies the SEM index-term types the extractors return (`eql_v3_internal.hmac_256`, -`eql_v3_internal.ore_block_256`, `eql_v3_internal.ope_cllw`) and the aggregate -state functions, so the generated SQL is entirely self-contained within the two +`eql_v3_internal.ore_block_256`, `eql_v3_internal.ope_cllw`), the aggregate +state functions, and every generated blocker function (e.g. +`eql_v3_internal."->"(...)` in `boolean_functions.sql`), so the generated SQL +is entirely self-contained within the two EQL-owned schemas. `tasks/build.sh` runs `cargo run -p eql-codegen` at the start of every `mise run -build`, so the generated SQL is never checked in. (The build first sweeps every -generated `*_{types,functions,operators,aggregates}.sql` under -`src/v3/scalars` so a type removed from `CATALOG` cannot leave orphans the -`src/**/*.sql` build glob would pick up; hand-written `*_extensions.sql` is -preserved by the name patterns.) +build`, regenerating the committed SQL under `src/v3/scalars/` in place. Orphan +removal lives inside codegen (`remove_generated_orphans`): after writing every +current file it prunes — marker-aware — any previously-generated `.sql` no +longer produced, so a type removed from `CATALOG` cannot leave orphans the +`src/**/*.sql` build glob would pick up; hand-written `*_extensions.sql` +carries no marker and always survives. `mise run codegen:parity` then asserts +the committed tree is unchanged by a regeneration (see "Generator tests and the +parity gate" below). Stages, in order (`generate_all` → `generate_type`): @@ -855,7 +886,12 @@ Stages, in order (`generate_all` → `generate_type`): `extractor_for_operator`, `role_for_terms`). 3. **Render.** `render_types_file`, `render_functions_file`, `render_operators_file`, and `render_aggregates_file` (the last only for - ordered domains) build the context structs in + ordered domains) render the column-domain surface; + `render_query_types_file`, `render_query_functions_file`, and + `render_query_operators_file` render the query-operand surface (the + `query_*` files, per term-bearing domain); and the `json_entry` cross + renderers emit the `json_entry__{functions,operators}.sql` pair (all + in `crates/eql-codegen/src/generate.rs`). They build the context structs in `crates/eql-codegen/src/context.rs` and render them through embedded **minijinja** templates (`crates/eql-codegen/templates/*.j2`, compiled in via `include_str!` — no runtime file IO). The structural shape of each declaration @@ -875,18 +911,37 @@ output for every catalog type from scratch. ### Generated outputs -For a type with `D` domains of which `A` are ordered, the generator writes `1 + -2D + A` SQL files into `src/v3/scalars//`. For `integer` (`D = 5`, `A = -3`): fourteen SQL files. The outputs are committed in place under +For a type with `D` domains, of which `A` are ordered and `Q` carry at least +one term, the generator writes `1 + 2D + A` **column-domain** SQL files, plus a +**query-operand** surface of `1 + 2Q` files, plus — for any family with an +`Ope`-bearing domain — two **`json_entry` cross** files, all into +`src/v3/scalars//`. For `integer` (`D = 5`, `A = 3`, `Q = 4`): +twenty-five SQL files (fourteen column-domain + nine query-operand + two +`json_entry`). The outputs are committed in place under `src/v3/scalars//` and regenerated at the start of every build (commit the regenerated SQL diff alongside any catalog change). +**Query-operand domains.** Every term-bearing column domain has a query twin, +`eql_v3.query__` (`Domain::query_name` in +`crates/eql-domains/src/spec.rs`): a term-only operand shape for typed query +parameters — the envelope `v`/`i` plus the domain's term keys, with **no** +ciphertext `c` (the `CHECK` requires `NOT (VALUE ? 'c')`). Query-operand +domains live in **`eql_v3`**, not `public`: they are never valid column types, +so dropping the EQL-owned schema can never drop an application column. The +`json_entry__*` pair is the cross-type surface comparing the extracted +SteVec leaf `public.eql_v3_json_entry` against the family's query operands — +ordering wrappers where the operand carries the deterministic `Ope` term, +blockers elsewhere. + | File | Content | | --------------------------------- | ---------------------------------------------------------------------------------------- | | `_types.sql` | Single idempotent `DO` block creating every domain; each `CHECK` pins the payload version (`VALUE->>'v' = '3'`) and required envelope/ciphertext/term keys; one `--! @brief` per domain | -| `_functions.sql` | One extractor per unique term, then 44 wrappers-or-blockers covering the surface | -| `_operators.sql` | 44 `CREATE OPERATOR` statements with planner metadata on supported ops | +| `_functions.sql` | One extractor per unique term, then 47 wrappers-or-blockers covering the surface | +| `_operators.sql` | 47 `CREATE OPERATOR` statements with planner metadata on supported ops | | `_aggregates.sql` | MIN/MAX state functions + `CREATE AGGREGATE`; emitted only for ordered domains | +| `query__types.sql` | `DO` block creating the `eql_v3.query_*` operand domains (term-only `CHECK`: `v`, `i`, term keys, `NOT (VALUE ? 'c')`) | +| `query___{functions,operators}.sql` | Wrappers/blockers and `CREATE OPERATOR` statements binding the column domain to its query-operand twin; one pair per term-bearing domain | +| `json_entry__{functions,operators}.sql` | The `public.eql_v3_json_entry` ↔ query-operand cross surface (ordering wrappers on `Ope`-bearing operands, blockers elsewhere) | Every file opens with the `-- AUTOMATICALLY GENERATED FILE.` marker (the project-wide marker `docs:validate` greps on to skip generated SQL — @@ -939,8 +994,8 @@ matrix from starting (the gate was removed — shards start after `build-archive Adding a new **term** is a bigger move than adding a type: edit the `Term` enum's `impl` methods, add `#[test]`s, add a `splinter.sh` entry for **each new name the term introduces** — its extractor *and* its comparison wrappers, plus any new SEM -constructor (adding `Bloom` required `match_term`, `contains`, `contained_by`, -and the SEM `bloom_filter`; adding `Ope` required `ord_term` and the SEM +constructor (adding `Bloom` required `match_term`, its `matches` fuzzy-match +wrapper, and the SEM `bloom_filter`; adding `Ope` required `ord_term` and the SEM `ope_cllw`) — and, because it changes the generated surface, regenerate and commit the affected SQL files under `src/v3/scalars//`. @@ -952,9 +1007,10 @@ regenerate and commit the affected SQL files under `text` **is** materialised through this generator. It is the worked example of an ordered, non-integer, unbounded scalar: it hand-writes its `impl ScalarType` + `OrderedScalar` (its `text` shape is read from the catalog `ScalarKind`, like -`date`'s temporal shape — there is no dispatch-list marker) with explicit -lexicographic `min`/`max` pivots instead of `::MIN`/`::MAX` and a real median -`mid_pivot()`. It is **`OrderedScalar` but not `SignedScalar`** — +`date`'s temporal shape — there is no dispatch-list marker); its boundary +pivots inherit the fixture-derived `min_pivot()`/`max_pivot()` defaults (§2 — +no impl overrides them), and it overrides only `mid_pivot()` with a real median +fixture. It is **`OrderedScalar` but not `SignedScalar`** — lexicographic text has no numeric origin, so it does not get the signed-only sign-boundary test, and the empty string is deliberately not a fixture (`""` encrypts to an empty ORE term; issue #262). `text` is also the first type to add @@ -1002,7 +1058,7 @@ What makes it storage-only: - **Generator: no changes needed.** The SQL generator already handles a zero-term, single-domain type — it emits exactly three files (`boolean_types.sql`, `boolean_functions.sql`, `boolean_operators.sql`; no `_aggregates.sql`, since no - ordered domain). All 44 functions are `plpgsql` blockers, all 44 `CREATE OPERATOR` statements back + ordered domain). All 47 functions are `plpgsql` blockers, all 47 `CREATE OPERATOR` statements back onto them: every comparison/containment/path operator reachable through domain fallback raises. The domain `CHECK` still pins `{v,i,c}` + `VALUE->>'v' = '3'`. - **Kind, not term.** Add a `ScalarKind` variant (`Bool`) with diff --git a/docs/reference/catalog-driven-architecture.md b/docs/reference/catalog-driven-architecture.md index da9114ed1..49c6fd5b2 100644 --- a/docs/reference/catalog-driven-architecture.md +++ b/docs/reference/catalog-driven-architecture.md @@ -16,7 +16,7 @@ gates guarantee the derived artifacts never drift from it. ```mermaid flowchart TD subgraph SOT["① SOURCE OF TRUTH — crates/eql-domains"] - CAT["CATALOG: &[DomainFamily]
(11 families: 10 scalar + jsonb)"] + CAT["CATALOG: &[DomainFamily]
(11 families: 10 scalar + json)"] FIX["FIXTURES: &[TypeFixtures]
(plaintext value lists)"] TERM["Term enum impls
(Hm / Ore / Bloom / Ope capabilities)"] CAT -.compile-time parity guard.- FIX @@ -70,13 +70,17 @@ Everything starts in `crates/eql-domains/src/lib.rs`: ```rust pub const CATALOG: &[DomainFamily] = &[ - INTEGER, SMALLINT, BIGINT, DATE, TIMESTAMP, NUMERIC, TEXT, BOOLEAN, REAL, DOUBLE, JSONB, + INTEGER, SMALLINT, BIGINT, DATE, TIMESTAMP, NUMERIC, TEXT, BOOLEAN, REAL, DOUBLE, JSON, ]; ``` Order is **load-bearing** — it drives generation order, inventory order, and snapshot order. -Ten of the eleven rows are `Shape::Scalar` families; the eleventh, `JSONB`, is the hand-written -SteVec family (see §2.3). Scalar-only consumers iterate `scalar_families()`, which filters `JSONB` out. +Ten of the eleven rows are `Shape::Scalar` families; the eleventh, `JSON`, is a **mixed** family — +three hand-written `Shape::SteVec` domains plus one generated `Shape::Scalar` storage domain +(`public.eql_v3_json`, rendered into `src/v3/scalars/json/` like any other storage-only domain; see §2.3). +Scalar-only consumers iterate `scalar_families()`, which filters `JSON` out wholesale (`is_scalar()` +is an `.all()`); the SQL/bindings generators instead iterate `families_with_scalar_domains()` and so +do render that storage domain. ### 2.1 The data model @@ -89,6 +93,7 @@ classDiagram class Domain { +name: &str // "", "eq", "ord", "ord_ore", "ord_ope", "match", "search" +terms: &[Term] + +shape: Shape // Scalar | SteVec } class Term { <> @@ -165,7 +170,7 @@ one of three catalog shapes. The invariant test `every_type_uses_a_known_domain_ iterates `scalar_families()`, not the full `CATALOG`, and accepts these current shapes plus two known-but-unused shapes (`eq-only` and `ordered+match`) so future scalar rows fail loudly if they drift into an unreviewed shape. The eleventh `CATALOG` family, -`jsonb`, is not scalar-shaped at all — see the note after the family table below. +`json`, is not scalar-shaped at all — see the note after the family table below. ```mermaid flowchart LR @@ -191,7 +196,7 @@ flowchart LR | `text` | Text | text-search (equality always routes through `Hm` — ORE is not equality-lossless for text) | | `boolean` | Bool | storage-only (2-value cardinality leak → no searchable index) | -**`jsonb` sits outside this classification.** It carries four domains. Three are +**`json` sits outside this classification.** It carries four domains. Three are `Shape::SteVec` — `public.eql_v3_json_search` (document), `public.eql_v3_json_entry` (one `sv` leaf), `eql_v3.query_json` (containment needle) — each with an empty flat `terms` list: capability lives *structurally* inside the payload (per-`sv`-leaf `hm` @@ -200,15 +205,15 @@ storage-only domain, `public.eql_v3_json` (bare `name: ""`, ciphertext-only `{v, no index terms). `Domain.name` (`"search"`/`"entry"`/`"query"`, and `""` for the storage domain) disambiguates which one a given domain is — see `Domain::rust_struct_name`. `scalar_families()` filters `CATALOG` down to families whose domains are *all* -`Shape::Scalar`, so `jsonb` never reaches `every_type_uses_a_known_domain_shape`, the +`Shape::Scalar`, so `json` never reaches `every_type_uses_a_known_domain_shape`, the ordered-scalar materializer (§3), or the scalar SQLx matrix. Its three SteVec domains are hand-written under `src/v3/json/` and their Rust structs in `crates/eql-bindings/src/v3/json.rs`; the generated storage domain's SQL is rendered into `src/v3/scalars/json/` and its struct into `crates/eql-bindings/src/v3/json_storage.rs` (via `families_with_scalar_domains()`) — for the SteVec domains, only inventory membership and `CATALOG` order are catalog-driven. This is also why the class diagram in §2.1 lists -`Jsonb` as a `ScalarKind` variant even though `jsonb` is not a `Shape::Scalar` family: -`ScalarKind` and `Shape` are independent axes, and `JSONB_FIXTURES` +`Jsonb` as a `ScalarKind` variant even though `json` is not a `Shape::Scalar` family: +`ScalarKind` and `Shape` are independent axes, and `JSON_FIXTURES` (`crates/eql-domains/src/fixtures/record.rs`) needs a `ScalarKind` purely for the `FIXTURES`/`CATALOG` parity machinery. @@ -279,17 +284,25 @@ discipline; `eql-codegen clean` exposes just the marker-aware SQL removal step. ### 3.1 SQL generation (`generate.rs` + `context.rs` + minijinja templates) -Per family, four renderers emit into `src/v3/scalars//`: +Per family, the renderers emit three surfaces into `src/v3/scalars//`: the +**column-domain** surface (types/functions/operators/aggregates), the **query-operand** +surface (`query_*` — one term-only `eql_v3.query__` twin per term-bearing +domain), and the **`json_entry` cross** surface (comparisons between the extracted +SteVec leaf `public.eql_v3_json_entry` and the family's query operands): ```mermaid flowchart TD F["DomainFamily"] --> T["render_types_file
→ <T>_types.sql"] + F --> QT["render_query_types_file
→ query_<T>_types.sql"] F --> PERDOM{"for each Domain"} PERDOM --> FN["render_functions_file
→ <T>_<dom>_functions.sql"] PERDOM --> OP["render_operators_file
→ <T>_<dom>_operators.sql"] PERDOM --> AG{"ORE-capable?"} AG -->|yes| AGG["render_aggregates_file
→ <T>_<dom>_aggregates.sql
(min/max)"] AG -->|no| SKIP["(no aggregates)"] + PERDOM --> QD{"term-bearing?"} + QD -->|yes| QFO["render_query_functions_file /
render_query_operators_file
→ query_<T>_<dom>_{functions,operators}.sql"] + F --> JE["json_entry cross renderers
→ json_entry_<T>_{functions,operators}.sql
(families with an Ope-bearing domain)"] ``` Each `*_functions.sql` mixes three entry kinds, selected per operator: @@ -355,7 +368,7 @@ distinctions become visible — e.g. `text_ord` lists `v i c hm op` (dual-term) ```text src/v3/scalars/ ├── functions.sql ← hand-written shared blocker helper (COMMITTED) -├── integer/ (14 generated files) +├── integer/ (25 generated files) │ ├── integer_types.sql (generated, committed) │ ├── integer_functions.sql (storage-only blockers) │ ├── integer_operators.sql (storage-only operator blockers) @@ -369,7 +382,18 @@ src/v3/scalars/ │ ├── integer_ord_aggregates.sql (min/max) │ ├── integer_ord_ope_functions.sql │ ├── integer_ord_ope_operators.sql -│ └── integer_ord_ope_aggregates.sql (min/max) +│ ├── integer_ord_ope_aggregates.sql (min/max) +│ ├── query_integer_types.sql (eql_v3.query_* operand domains, term-only) +│ ├── query_integer_eq_functions.sql ┐ +│ ├── query_integer_eq_operators.sql │ one functions/operators pair +│ ├── query_integer_ord_functions.sql │ per term-bearing domain +│ ├── query_integer_ord_operators.sql │ +│ ├── query_integer_ord_ore_functions.sql │ +│ ├── query_integer_ord_ore_operators.sql │ +│ ├── query_integer_ord_ope_functions.sql │ +│ ├── query_integer_ord_ope_operators.sql ┘ +│ ├── json_entry_integer_functions.sql (json_entry ↔ query-operand cross surface) +│ └── json_entry_integer_operators.sql │ (integer_extensions.sql ← hand-written, COMMITTED, only if present) └── ... ``` @@ -388,13 +412,16 @@ flowchart TD MOD["mod.rs
module doc: float-NaN + bool caveats"] DT["domain_type.rs
DomainType trait"] TR["terms.rs
Ciphertext/Hmac256/
OreBlock256/BloomFilter"] + JSRS["json.rs
SteVec payload types"] LIB["lib.rs
SchemaVersion/Identifier"] end - subgraph gen["GENERATED (@generated)"] - I4["integer.rs / text.rs / ... (10 files)"] + subgraph gen["GENERATED (@generated) — 14 files"] + I4["integer.rs / text.rs / ... (10 family files)
+ json_storage.rs"] + PAY["payload.rs / query_payload.rs
DomainPayload / QueryPayload enums"] INV["inventory.rs — all()"] end DT --> I4 + DT --> PAY TR --> I4 LIB --> I4 MOD --> gen @@ -440,17 +467,17 @@ operator blocked). ```mermaid flowchart LR - RS["Rust structs
#[derive(TS, JsonSchema)]"] -->|ts-rs export
via cargo test -p eql-bindings| TS["crates/eql-bindings/bindings/v3/IntegerEq.ts
(63 files)"] - RS -->|schemars via tests/export.rs
injects $id| JS["crates/eql-bindings/schema/v3/integer_eq.json
(51 files)"] + RS["Rust structs
#[derive(TS, JsonSchema)]"] -->|ts-rs export
via cargo test -p eql-bindings| TS["crates/eql-bindings/bindings/v3/IntegerEq.ts
(104 files)"] + RS -->|schemars via tests/export.rs
injects $id| JS["crates/eql-bindings/schema/v3/integer_eq.json
(92 files)"] ``` - **TypeScript:** one `.ts` per domain, importing co-located term types; newtypes become - primitive aliases (`export type Ciphertext = string;`, `export type SchemaVersion = 2;`). + primitive aliases (`export type Ciphertext = string;`, `export type SchemaVersion = 3;`). Doc comments survive from Rust. - **JSON Schema:** JSON Schema 2020-12 (schemars 1.x), `additionalProperties: false`, `$id` injected at export (`https://schemas.cipherstash.com/eql/v3/integer_eq.json`), term types as reusable `$defs`. `BloomFilter` and `SchemaVersion` have **manual** `JsonSchema` - impls (bounded `i16[]`, `const: 2`) that derives can't express. + impls (bounded `i16[]`, `const: 3`) that derives can't express. > **Per-field vs per-struct docs:** there are *no* per-field docs on generated structs. > Per-term semantics live on the shared term newtypes in `terms.rs` (flowing into TS term diff --git a/docs/reference/database-indexes.md b/docs/reference/database-indexes.md index 274f35ce3..d1dd05191 100644 --- a/docs/reference/database-indexes.md +++ b/docs/reference/database-indexes.md @@ -24,7 +24,7 @@ Each capability has one canonical functional-index recipe. Type the column as th ```sql -- Equality (hash index on the eq_term extractor) — the hm-carrying domains: --- public._eq / text_ord / text_ord_ore / text_search / text_search_ore +-- public._eq / text_ord / text_ord_ope / text_ord_ore / text_search / text_search_ore CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(encrypted_email)); @@ -32,7 +32,7 @@ CREATE INDEX users_email_eq CREATE INDEX events_at_ord ON events USING btree (eql_v3.ord_term(encrypted_at)); --- Text match (bloom-filter fuzzy match `@@` — GIN on the match_term extractor) — public.eql_v3_text_match / text_search +-- Text match (bloom-filter fuzzy match `@@` — GIN on the match_term extractor) — public.eql_v3_text_match / text_search / text_search_ore CREATE INDEX users_name_match ON users USING gin (eql_v3.match_term(encrypted_name)); @@ -41,13 +41,13 @@ ANALYZE users; > **No operator class on a column or domain.** `eql_v3` deliberately does **not** ship an `encrypted_operator_class`. Operators resolve against the domain's `jsonb` base type, so an opclass on the column would bypass the encrypted surface. Always index through the extractor. (This also means no superuser is required to *query* — functional indexes work on Supabase and managed PostgreSQL.) -> **ORE requires a superuser *install*; OPE does not.** The `ord_term` btree recipe above needs nothing installed: the default `_ord` domains order via CLLW-OPE, and `eql_v3.ord_term` returns a `bytea`-backed type whose *native* btree operator class the planner already has. The block-ORE domains (`_ord_ore`, `text_search_ore`) instead depend on the operator class the installer creates for the ORE term type — and `CREATE OPERATOR CLASS` requires superuser. On platforms whose installer role is not superuser (cloud-hosted Supabase, most managed Postgres), the installer detects this and **disables the ORE-carrying domains** (`_ord_ore`, `text_search_ore`, and their `eql_v3.query_*` twins): using one raises `feature_not_supported` with a `HINT` naming the alternatives (see [U-003](../upgrading/v3.0.md#u-003-non-superuser-installs-disable-the-ore-backed-domains)). On a superuser install, index an `_ord_ore` column through its own extractor: +> **ORE requires an *install* privilege; OPE does not.** The `ord_term` btree recipe above needs nothing installed: the default `_ord` domains order via CLLW-OPE, and `eql_v3.ord_term` returns a `bytea`-backed type whose *native* btree operator class the planner already has. The block-ORE domains (`_ord_ore`, `text_search_ore`) instead depend on the operator class the installer creates for the ORE term type — and `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL. Whether a managed platform allows it is per-platform, not a blanket rule: AWS RDS and Aurora PostgreSQL allow it for the master user, while cloud-hosted Supabase, Cloud SQL, and Azure Flexible Server refuse it (see [Install privileges](./permissions.md#install-privileges) for the details and sources). Where the install role can't create the opclass, the installer detects this and **disables the ORE-carrying domains** (`_ord_ore`, `text_search_ore`, and their `eql_v3.query_*` twins): using one raises `feature_not_supported` with a `HINT` naming the alternatives (see [U-003](../upgrading/v3.0.md#u-003-non-superuser-installs-disable-the-ore-backed-domains)). Where the opclass installed, index an `_ord_ore` column through its own extractor: > > ```sql > CREATE INDEX events_at_ord_ore ON events USING btree (eql_v3.ord_term_ore(encrypted_at)); > ``` -> **Equality on the ordering domains splits on term injectivity.** The numeric-and-time ordering terms (OPE and ORE alike) are **injective** — distinct plaintexts produce distinct terms — so equality can ride the ordering term: those `_ord` / `_ord_ore` domains carry no `hm`, there is **no `eq_term` overload** for them, and `eql_v3.eq` inlines to an ordering-term comparison (`ord_term(a) = ord_term(b)`). One ordering btree serves `=`, range, and `ORDER BY` — do not create an `eq_term` index on a numeric `_ord` column; the overload does not exist. Text ordering terms are **non-injective** and cannot be relied on for equality, so the text ordering domains (`text_ord`, `text_ord_ore`, and the `text_search` variants) also carry `hm` and answer `=` via `eq_term` — give those columns an equality index alongside the ordering one. +> **Equality on the ordering domains splits on term injectivity.** The numeric-and-time ordering terms (OPE and ORE alike) are **injective** — distinct plaintexts produce distinct terms — so equality can ride the ordering term: those `_ord` / `_ord_ope` / `_ord_ore` domains carry no `hm`, there is **no `eq_term` overload** for them, and `eql_v3.eq` inlines to an ordering-term comparison (`ord_term(a) = ord_term(b)`). One ordering btree serves `=`, range, and `ORDER BY` — do not create an `eq_term` index on a numeric `_ord` column; the overload does not exist. Text ordering terms are **non-injective** and cannot be relied on for equality, so the text ordering domains (`text_ord`, `text_ord_ope`, `text_ord_ore`, and the `text_search` variants) also carry `hm` and answer `=` via `eq_term` — give those columns an equality index alongside the ordering one. ### When to Create Indexes @@ -55,7 +55,7 @@ Create indexes on encrypted columns when: - The table has a significant number of rows (typically > 1000). - You frequently query the column by the matching operator. -- The column is typed as a variant that carries the required term (`_eq` for equality, `_ord` for range/ordering, `text_match` for bloom fuzzy match `@@`). +- The column is typed as a variant that carries the required term (`_eq` for equality, `_ord` for range/ordering, `text_match` for bloom fuzzy match `@@`). On the numeric-and-time `_ord` domains the ordering variant serves `=` too — no `_eq` twin needed; see the injectivity note above. --- @@ -81,7 +81,7 @@ For PostgreSQL to use a functional index on an encrypted column, **all** of thes Capability travels in the payload, chosen by the encryption client and reflected in the column's domain variant: -- **Equality** needs the domain's equality-serving term. On the `hm`-carrying domains (`public._eq`, `public.eql_v3_text_ord`, `public.eql_v3_text_ord_ore`, `public.eql_v3_text_search`, `public.eql_v3_text_search_ore`) that is `hm` (hmac_256), driven through `eql_v3.eq_term`. On the numeric-and-time `_ord` / `_ord_ore` domains it is the **ordering term itself** (`op` / `ob`) — injective for those types, so `eql_v3.eq` compares ordering terms and the ordering index serves equality (see [Creating Indexes](#creating-indexes)). +- **Equality** needs the domain's equality-serving term. On the `hm`-carrying domains (`public._eq`, `public.eql_v3_text_ord`, `public.eql_v3_text_ord_ope`, `public.eql_v3_text_ord_ore`, `public.eql_v3_text_search`, `public.eql_v3_text_search_ore`) that is `hm` (hmac_256), driven through `eql_v3.eq_term`. On the numeric-and-time `_ord` / `_ord_ope` / `_ord_ore` domains it is the **ordering term itself** (`op` / `ob`) — injective for those types, so `eql_v3.eq` compares ordering terms and the ordering index serves equality (see [Creating Indexes](#creating-indexes)). - **Range / ordering** needs an ordering term — `op` (ope_cllw) on `public._ord` / `_ord_ope` / `public.eql_v3_text_search`, or `ob` (ore_block_256) on `public._ord_ore` / `public.eql_v3_text_search_ore`. - **Text match** (`@@`) needs a `bf` (bloom_filter) term — `public.eql_v3_text_match`, `public.eql_v3_text_search`, or `public.eql_v3_text_search_ore`. @@ -98,19 +98,21 @@ The comparison value must resolve to the encrypted operator, not the native `jso ```sql -- ✓ resolves the encrypted operator → uses the index WHERE encrypted_email = $1; -WHERE encrypted_email = $1::public.eql_v3_text_eq; +WHERE encrypted_email = $1::eql_v3.query_text_eq; -- ✗ a bare jsonb literal falls through to native jsonb semantics WHERE encrypted_email = '{"hm":"abc"}'::jsonb; ``` +Cast an explicit operand to the matching **query-operand domain** (`eql_v3.query__`), not the column's storage domain: query payloads are term-only, and the storage domains' CHECK requires the ciphertext key `c` that query payloads deliberately omit. + --- ## Query Patterns That Use Indexes ### Equality Queries -A column typed as an `hm`-carrying domain (`public._eq`, `text_ord`, `text_ord_ore`, or a `text_search` variant) with a hash index on `eql_v3.eq_term(col)`: +A column typed as an `hm`-carrying domain (`public._eq`, `text_ord`, `text_ord_ope`, `text_ord_ore`, or a `text_search` variant) with a hash index on `eql_v3.eq_term(col)`: ```sql CREATE INDEX users_email_eq ON users USING hash (eql_v3.eq_term(encrypted_email)); @@ -121,7 +123,7 @@ SELECT * FROM users WHERE encrypted_email = $1; -- Index Cond: (eql_v3.eq_term(encrypted_email) = eql_v3.eq_term($1)) ``` -On a numeric-and-time `_ord` / `_ord_ore` column there is no `eq_term` — `=` inlines to the ordering extractor instead, so the same equality predicate engages the ordering btree from the [range recipe below](#range-queries-and-order-by): +On a numeric-and-time `_ord` / `_ord_ope` / `_ord_ore` column there is no `eq_term` — `=` inlines to the ordering extractor instead, so the same equality predicate engages the ordering btree from the [range recipe below](#range-queries-and-order-by): ```sql SELECT * FROM events WHERE encrypted_at = $1; @@ -140,7 +142,7 @@ ANALYZE events; `eql_v3.ord_term` returns `eql_v3_internal.ope_cllw`, a domain over `bytea`, so this btree resolves to `bytea_ops` — PostgreSQL's **default** operator class for the base type. Nothing to install, no privilege needed, and the opfamily already contains the `<` `<=` `>` `>=` the planner needs. -> **Why this matters on managed PostgreSQL.** The `_ord_ore` path depends on a hand-written btree operator class for the `eql_v3_internal.ore_block_256` *composite*, created by a `DO` block that is silently skipped when the installing role is not superuser. If that opclass is absent, `CREATE INDEX … btree (eql_v3.ord_term_ore(col))` does **not** fail — PostgreSQL binds `record_ops` instead. The index builds, occupies space, and never engages, because the ORE comparison operators are not members of `record_ops`. A silently useless index is worse than a rejected one. `_ord` has no such failure mode. +> **Why this matters on managed PostgreSQL.** The `_ord_ore` path depends on a hand-written btree operator class for the `eql_v3_internal.ore_block_256` *composite*, created by a `DO` block that is skipped (with only a `NOTICE`, which most migration tooling swallows) when the installing role cannot create operator classes. If that opclass is absent, `CREATE INDEX … btree (eql_v3.ord_term_ore(col))` does **not** fail — PostgreSQL binds `record_ops` instead. The index builds, occupies space, and never engages, because the ORE comparison operators are not members of `record_ops`. A silently useless index is worse than a rejected one. `_ord` has no such failure mode. > > Check which opclass an existing index actually bound: > @@ -185,6 +187,8 @@ SELECT eql_v3.eq_term(encrypted_email), count(*) Why the raw column does not scale: `GROUP BY col` uses the entire encrypted payload (1–2 KB per row) as the hash key. PostgreSQL estimates a hash table far larger than the default `work_mem` (4 MB), refuses `HashAggregate`, and falls back to `GroupAggregate` — sorting kilobyte-sized rows and spilling to disk. The `eql_v3.eq_term(col)` key is a small deterministic term, so the hash table fits in `work_mem` and the planner picks `HashAggregate` reliably — without any deployment-wide tuning. If you cannot rewrite the query (an ORM grouping the raw column), bumping `work_mem` to fit the estimated hash table is the rescue knob, but the extractor form is the design. +Pick the extractor the domain actually has: `eq_term` exists only on the `hm`-carrying domains. On a numeric-and-time `_ord` / `_ord_ope` / `_ord_ore` column, group on the ordering extractor instead — `GROUP BY eql_v3.ord_term(col)` (or `ord_term_ore(col)`). The ordering term is injective for those types, so it is an exact grouping key, and the column's ordering btree covers it. + ### Field-level ordering index (ste_vec elements) Entry-to-entry `=` / `<>` and exact `GROUP BY` / `DISTINCT` on extracted @@ -202,7 +206,7 @@ For document-level containment (`@>`) on `public.eql_v3_json_search` columns, us ```sql CREATE INDEX orders_data_gin - ON orders USING gin (eql_v3.to_ste_vec_query(data_encrypted)::jsonb jsonb_path_ops); + ON orders USING gin ((eql_v3.to_ste_vec_query(data_encrypted)::jsonb) jsonb_path_ops); ANALYZE orders; SELECT * FROM orders WHERE data_encrypted @> $1::eql_v3.query_json; @@ -248,7 +252,7 @@ It is the single highest-leverage knob for build time. On a managed deployment w ### Index type decides whether the build scales -For *query* performance the access method is settled by capability (`hash` for equality, `btree` for ORE, `GIN` for bloom / ste_vec). For *build* performance at scale they are not equivalent: +For *query* performance the access method is settled by capability (`hash` or `btree` on `eq_term` for equality, `btree` on the ordering extractor — OPE and ORE alike, `GIN` for bloom / ste_vec). For *build* performance at scale they are not equivalent: | Access method | Build algorithm | Scales past cache? | Parallel build? | | ------------- | ------------------------------------------------- | ------------------ | --------------- | @@ -258,7 +262,7 @@ For *query* performance the access method is settled by capability (`hash` for e A hash build scatters consecutive heap rows to random buckets; once the index outgrows `shared_buffers` + OS cache it becomes random-I/O-bound and cannot be parallelised. A btree build sorts first, then writes sequentially across parallel workers. -**For equality functional indexes on large tables, prefer `btree` over `hash`.** `eql_v3.eq_term(col)` — and the field-level `eql_v3.eq_term(col -> ''::text)` — return small deterministic terms; a btree on them serves `=` exactly as well as a hash index, with no query-side cost, and the build goes from pathological to routine: +**For equality functional indexes on large tables, prefer `btree` over `hash`.** `eql_v3.eq_term(col)` returns a small deterministic term; a btree on it serves `=` exactly as well as a hash index, with no query-side cost, and the build goes from pathological to routine: ```sql CREATE INDEX … USING btree (eql_v3.eq_term(col)); -- large tables @@ -297,7 +301,7 @@ The first move on a slow EQL query is `EXPLAIN (COSTS OFF)`. Look for: - **`Bitmap Index Scan on `** — same, for set-style predicates (`@>`). ✓ - **`Index Cond:`** referencing the extractor (`eql_v3.eq_term(…)`, `eql_v3.ord_term(…)`) — the inlined predicate matched the index. ✓ - **`Seq Scan`** — no index used. Investigate. -- **`Filter:` showing the raw operator** (`col < '…'`) — inlining did not happen. Usual causes: a pinned `search_path` on a customised function (`\df+` shows `proconfig`), a `plpgsql` body where a `sql` one is expected, or the planner judging another plan cheaper. +- **`Filter:` showing the raw operator** (`col < '…'`) — inlining did not happen. Usual causes: a pinned `search_path` on a customised function (`\sf ` shows any `SET search_path` clause), a `plpgsql` body where a `sql` one is expected, or the planner judging another plan cheaper. - **`Sort` node above an Index Scan** — natural-form `ORDER BY`; expected for that shape. Switch the sort key to the column's ordering extractor to eliminate it. Once a plan looks right, repeat with `EXPLAIN ANALYZE` to measure actual timings. @@ -311,11 +315,12 @@ Once a plan looks right, repeat with `EXPLAIN ANALYZE` to measure actual timings 1. **Verify the value carries the term.** Equality needs `hm` on the `hm`-carrying domains (the ordering term serves it on numeric-and-time `_ord` / `_ord_ore`), range needs `op` (or `ob` on an `_ord_ore` column), containment needs `bf`: ```sql SELECT encrypted_email::jsonb ? 'hm' AS has_hmac, + encrypted_email::jsonb ? 'op' AS has_ope, encrypted_email::jsonb ? 'ob' AS has_ore_block, encrypted_email::jsonb ? 'bf' AS has_bloom FROM users LIMIT 1; ``` -2. **Verify the operand is typed** (`$1::public.eql_v3_text_eq`, not `$1::jsonb`). +2. **Verify the operand is typed** (`$1::eql_v3.query_text_eq`, not `$1::jsonb` — and not the storage domain `public.eql_v3_text_eq`, whose CHECK requires the ciphertext key query payloads omit). 3. **Recreate the index** if the column's terms changed after the index was built. 4. **Run `ANALYZE`** — very small tables may still choose a sequential scan, which is correct. diff --git a/docs/reference/eql-functions.md b/docs/reference/eql-functions.md index b3b0efc8b..feb3d62e4 100644 --- a/docs/reference/eql-functions.md +++ b/docs/reference/eql-functions.md @@ -11,6 +11,7 @@ A reference for the functions and operators EQL exposes for querying encrypted d - [Index Term Extraction](#index-term-extraction) - [Encrypted JSON (`public.eql_v3_json_search`)](#encrypted-json-publiceql_v3_json_search) - [Aggregate Functions](#aggregate-functions) +- [Diagnostics](#diagnostics) --- @@ -20,21 +21,23 @@ EQL overloads standard PostgreSQL operators on the encrypted-domain types. Type ### Equality — `=` `<>` -On `public._eq`, `public._ord` / `_ord_ore`, and `public.eql_v3_text_search` (carry an `hm` term): +On `public._eq`, `public._ord` / `_ord_ope` / `_ord_ore`, and `public.eql_v3_text_search` / `_search_ore`. Equality compares the `hm` term where the domain carries one (the `_eq` domains and the text ordering/search domains) and otherwise compares the ordering term directly — the numeric-and-time ordering terms are injective, so those domains carry no `hm` and `eql_v3.eq` inlines to an ordering-term comparison: ```sql SELECT * FROM users WHERE encrypted_email = $1; -SELECT * FROM users WHERE encrypted_email = $1::public.eql_v3_text_eq; +SELECT * FROM users WHERE encrypted_email = $1::eql_v3.query_text_eq; SELECT * FROM users WHERE encrypted_email <> $1; ``` +(Explicit casts use the **query-operand** domain `eql_v3.query__` — query payloads are term-only and fail the storage domain's CHECK, which requires the ciphertext key `c`.) + ### Range — `<` `<=` `>` `>=` -On `public._ord` / `_ord_ope` / `_ord_ore` and `public.eql_v3_text_search` (carry an ordering term): +On `public._ord` / `_ord_ope` / `_ord_ore` and `public.eql_v3_text_search` / `_search_ore` (carry an ordering term): ```sql -SELECT * FROM events WHERE encrypted_at < $1::public.eql_v3_timestamp_ord; -SELECT * FROM events WHERE encrypted_at >= $1::public.eql_v3_timestamp_ord; +SELECT * FROM events WHERE encrypted_at < $1::eql_v3.query_timestamp_ord; +SELECT * FROM events WHERE encrypted_at >= $1::eql_v3.query_timestamp_ord; -- Ordering (write the sort key as the extractor to engage the index — see Database Indexes) -- `encrypted_at` is a `timestamp_ord` column, so its extractor is `ord_term`. @@ -47,7 +50,7 @@ On `public.eql_v3_text_match` / `public.eql_v3_text_search` / `public.eql_v3_text_search_ore` (carry a `bf` bloom term). This is **probabilistic ngram-bloom matching** (`eql_v3.matches`), not SQL `LIKE`, not JSONB containment, and not the containment operators — `@>` / `<@` **raise** on these domains: ```sql -SELECT * FROM docs WHERE encrypted_content @@ $1::public.eql_v3_text_match; +SELECT * FROM docs WHERE encrypted_content @@ $1::eql_v3.query_text_match; ``` `LIKE` / `ILIKE` (`~~` / `~~*`) are **not** part of the `eql_v3` surface — use `@@`. @@ -63,9 +66,9 @@ SELECT * FROM docs WHERE encrypted_content @@ $1::public.eql_v3_text_match; For environments that cannot use custom operators (e.g. some managed platforms), each operator has a function form, generated per domain variant. They take the same domain types as the operators above: ```sql -eql_v3.eq(a, b) -- = (on _eq / _ord / text_search) +eql_v3.eq(a, b) -- = (on _eq / _ord / _ord_ope / _ord_ore / text_search / text_search_ore) eql_v3.neq(a, b) -- <> -eql_v3.lt(a, b) -- < (on _ord / _ord_ore / text_search) +eql_v3.lt(a, b) -- < (on _ord / _ord_ope / _ord_ore / text_search / text_search_ore) eql_v3.lte(a, b) -- <= eql_v3.gt(a, b) -- > eql_v3.gte(a, b) -- >= @@ -77,8 +80,8 @@ JSON document containment has its own function forms (`eql_v3.jsonb_contains` / **Example:** ```sql -SELECT * FROM users WHERE eql_v3.eq(encrypted_email, $1::public.eql_v3_text_eq); -SELECT * FROM events WHERE eql_v3.lt(encrypted_at, $1::public.eql_v3_timestamp_ord); +SELECT * FROM users WHERE eql_v3.eq(encrypted_email, $1::eql_v3.query_text_eq); +SELECT * FROM events WHERE eql_v3.lt(encrypted_at, $1::eql_v3.query_timestamp_ord); ``` There are no `like` / `ilike` function forms — text matching is `eql_v3.matches` (`@@`) on a `text_match` value. @@ -122,7 +125,7 @@ See [json-support.md](./json-support.md). ## Encrypted JSON (`public.eql_v3_json_search`) -The full encrypted-JSONB function surface — containment, `->` / `->>`, `eql_v3.jsonb_path_query` / `_first` / `_exists`, `eql_v3.jsonb_array_length` / `_elements` / `_elements_text`, `eql_v3.to_ste_vec_query`, `eql_v3.ste_vec_contains`, and the GIN helpers — is documented in **[EQL with JSON and JSONB](./json-support.md)**. +The full encrypted-JSONB function surface — containment, `->` / `->>`, `eql_v3.jsonb_path_query` / `_first` / `_exists`, `eql_v3.jsonb_array_length` / `_elements`, `eql_v3.to_ste_vec_query`, `eql_v3.ste_vec_contains`, the envelope accessors `eql_v3.meta_data` / `eql_v3.ciphertext` / `eql_v3.selector`, and the GIN helpers — is documented in **[EQL with JSON and JSONB](./json-support.md)**. (`eql_v3.jsonb_array_elements_text` was removed in 3.0 — a bare-ciphertext stream is no longer independently decryptable; use `eql_v3.jsonb_array_elements`.) --- @@ -130,7 +133,7 @@ The full encrypted-JSONB function surface — containment, `->` / `->>`, `eql_v3 ### `eql_v3.min()` / `eql_v3.max()` (per-domain) -Returns the minimum or maximum encrypted value on an ordered encrypted-domain column. Defined per ord-capable variant of every scalar type (`public._ord`, `public._ord_ore`); the input type selects the aggregate via PostgreSQL's overload resolution. +Returns the minimum or maximum encrypted value on an ordered encrypted-domain column. Defined per ord-capable variant of every scalar type (`public._ord`, `public._ord_ope`, `public._ord_ore`, plus `public.eql_v3_text_search` / `_search_ore`); the input type selects the aggregate via PostgreSQL's overload resolution. ```sql -- integer — generated for every ordered variant of every scalar type. @@ -161,6 +164,13 @@ SELECT eql_v3.min(price_jsonb::public.eql_v3_integer_ord) FROM products; --- +## Diagnostics + +- **`eql_v3.version() RETURNS text`** — the installed EQL release version, baked in at build time. First check when behaviour doesn't match the docs: `SELECT eql_v3.version();` +- **`eql_v3.lints() RETURNS SETOF record (severity, category, object_name, message)`** — installation self-checks (missing opclass, misconfigured objects, …): `SELECT * FROM eql_v3.lints();` + +--- + ## See Also - [EQL Configuration Tutorial](../tutorials/proxy-configuration.md) — setting up encrypted columns end to end. diff --git a/docs/reference/json-support.md b/docs/reference/json-support.md index eae1ab85d..2c633e8e1 100644 --- a/docs/reference/json-support.md +++ b/docs/reference/json-support.md @@ -8,6 +8,7 @@ EQL encrypts, decrypts, and searches JSON / JSONB documents using structured enc - [Typed operands (important)](#typed-operands-important) - [Querying `public.eql_v3_json_search`](#querying-publiceql_v3_json_search) - [Containment queries (`@>`, `<@`)](#containment-queries--) + - [Selector-with-constraint range queries](#selector-with-constraint-range-queries-index-accelerated) - [Field extraction (`jsonb_path_query`)](#field-extraction-jsonb_path_query) - [JSON path operators (`->`, `->>`)](#json-path-operators----) - [Array operations](#array-operations) @@ -42,14 +43,20 @@ Always give the operand a known type: ```sql -- ✅ correct — typed operand resolves to the eql_v3 operator -WHERE doc -> 'email'::text = $1 WHERE doc @> $1::eql_v3.query_json -WHERE doc -> $1 -- a text parameter (the CipherStash Proxy interface) - --- ⚠ wrong — bare untyped literal resolves to native jsonb -> text, returns NULL -WHERE doc -> 'email' +WHERE doc -> 'age_selector'::text > $1::eql_v3.query_integer_ord +WHERE doc -> $1::text > $2::eql_v3.query_integer_ord -- selector bound as a text + -- parameter (the CipherStash + -- Proxy interface) + +-- ⚠ wrong — the bare untyped selector resolves to native jsonb -> text, so the +-- extraction is a plain root-key lookup (NULL) and the comparison falls through +-- to native jsonb semantics instead of the encrypted operator +WHERE doc -> 'email' > $1::eql_v3.query_integer_ord ``` +(Note there is no entry-level `=` — exact field equality goes through containment; see [Grouping data](#grouping-data) for why, and the range example above for what extracted entries *do* support.) + This is **intrinsic to the domain type-kind**, not a bug: the only way to remove it would be to make `public.eql_v3_json_search` a base type (losing free `jsonb` interop). The CipherStash Proxy always passes typed parameters, so applications routing through the Proxy are unaffected; the caveat matters only for hand-written ad-hoc SQL. ## Querying `public.eql_v3_json_search` @@ -83,7 +90,7 @@ For large tables, back containment with a GIN index. The typed `@>` overload inl ```sql CREATE INDEX examples_json_gin - ON examples USING gin (eql_v3.to_ste_vec_query(encrypted_json)::jsonb jsonb_path_ops); + ON examples USING gin ((eql_v3.to_ste_vec_query(encrypted_json)::jsonb) jsonb_path_ops); ANALYZE examples; SELECT * FROM examples WHERE encrypted_json @> $1::eql_v3.query_json; @@ -108,7 +115,7 @@ SELECT eql_v3.jsonb_path_exists(encrypted_json, 'abc123def456...') FROM examples ### JSON path operators (`->`, `->>`) -`->` returns the matched entry as an `public.eql_v3_json_entry`; `->>` returns it serialized as `text` (ciphertext JSON, not decrypted plaintext). The selector operand must be typed: +`->` returns the matched entry as a `public.eql_v3_json_entry`; `->>` returns it serialized as `text` (ciphertext JSON, not decrypted plaintext). The selector operand must be typed: ```sql -- Field access by selector (returns public.eql_v3_json_entry) @@ -205,6 +212,8 @@ separate equality-indexed scalar column when server-side grouping is required. - **`eql_v3.ste_vec(val jsonb) RETURNS jsonb[]`** — extracts the ste_vec index array from an encrypted payload. - **`eql_v3.ste_vec_contains(a public.eql_v3_json_search, b public.eql_v3_json_search) RETURNS boolean`** — true if every selector in `b` is present in `a` (selector-subset containment); backs the `@>` operator. +- **`eql_v3.jsonb_contains(a jsonb, b jsonb)` / `eql_v3.jsonb_contained_by(a jsonb, b jsonb)`** — function-form containment entrypoints for platforms that cannot type an operator call (Supabase PostgREST RPC, for example); same result as `@>` / `<@` (a parity test pins this). +- **`eql_v3.jsonb_array(val jsonb) RETURNS jsonb[]`** — function-form array accessor for the same platforms. - **`eql_v3.to_ste_vec_query(val public.eql_v3_json_search) RETURNS eql_v3.query_json`** — the GIN-indexable query shape `@>` inlines to. - **`eql_v3.meta_data(val jsonb)`** — envelope accessor: returns `{i, v, h}` (the key header `h` included so an extracted entry can be decrypted). **`eql_v3.ciphertext(val jsonb)`** — returns the `c` field; on the ste_vec surface `c` is raw AEAD output, decryptable only together with the entry's `s` and the document `h`. **`eql_v3.selector(val jsonb)` / `(entry public.eql_v3_json_entry)`** — selector accessor. @@ -225,11 +234,13 @@ separate equality-indexed scalar column when server-side grouping is required. - **`eql_v3.ord_term(entry public.eql_v3_json_entry)`** — ordering term (backs `<` … `>=`); returns SQL `NULL` when the leaf carries no `op` term. - **`eql_v3.min(public.eql_v3_json_entry)` / `eql_v3.max(...)`** — MIN / MAX over an extracted ordered leaf. -For GIN-indexable JSONB containment, see [GIN Indexes for JSONB Containment](./database-indexes.md#gin-indexes-for-jsonb-containment) (`eql_v3.to_ste_vec_query(col)::jsonb jsonb_path_ops`). +For GIN-indexable JSONB containment, see [GIN Indexes for JSONB Containment](./database-indexes.md#gin-indexes-for-jsonb-containment) (`USING gin ((eql_v3.to_ste_vec_query(col)::jsonb) jsonb_path_ops)`). ### Blocked operators -The native `jsonb` operators `?`, `?|`, `?&`, `@?`, `@@`, `#>`, `#>>`, `-`, `#-`, `||`, root-document `=` `<>` `<` `<=` `>` `>=`, single-entry containment, and entry-to-entry `=` / `<>` are **blocked** — they `RAISE` rather than running plaintext-jsonb or lossy ordering-term equality semantics. Use document containment for exact equality and extracted-entry ordering only for ranges. +The native `jsonb` operators `?`, `?|`, `?&`, `@?`, `@@`, `#>`, `#>>`, `-`, `#-`, `||`, root-document `=` `<>` `<` `<=` `>` `>=`, single-entry containment, containment against a native `jsonb` operand (`doc @> jsonb` and every other operand mix, both directions), and entry-to-entry `=` / `<>` are **blocked** — they `RAISE` rather than running plaintext-jsonb or lossy ordering-term equality semantics. Use document containment for exact equality and extracted-entry ordering only for ranges. + +> **`eql_v3_json_search` vs `eql_v3_json`:** this page is about the *queryable* document domain. A sibling `public.eql_v3_json` domain also exists — **storage only**, every operator a blocker. Use it for encrypted JSON you never query in the database; use `eql_v3_json_search` when you need containment/selector queries. ## How ste_vec indexing works diff --git a/docs/reference/permissions.md b/docs/reference/permissions.md index 147e88d5d..7d4c3893d 100644 --- a/docs/reference/permissions.md +++ b/docs/reference/permissions.md @@ -18,6 +18,67 @@ Two consequences follow from standard PostgreSQL behaviour: A deployment that exposes EQL to non-owner roles must grant access explicitly. +## Install privileges + +Installing EQL v3 requires only `CREATE` on the target database and schemas — +**no superuser** — with one exception: the block-ORE btree operator +class/family (`eql_v3_internal.ore_block_256_operator_class`). `CREATE +OPERATOR CLASS` / `CREATE OPERATOR FAMILY` are superuser-gated commands in +stock PostgreSQL (every supported version — the [`CREATE OPERATOR +CLASS`](https://www.postgresql.org/docs/current/sql-createopclass.html) page: +"Presently, the creating user must be a superuser"), and the privilege is not +grantable. + +The installer handles this gracefully: the opclass is created inside a `DO` +block that catches `insufficient_privilege` (SQLSTATE 42501) and skips with a +`NOTICE`, after which the ORE-backed domains (`_ord_ore`, `text_search_ore`, +and their `eql_v3.query_*` twins) are **disabled** — using one raises +`feature_not_supported` with a `HINT` naming the alternatives (see +[U-003](../upgrading/v3.0.md#u-003-non-superuser-installs-disable-the-ore-backed-domains)). +Everything else — the domains, operators, extractors, aggregates, and every +functional-index recipe except the ORE one — installs and works without +superuser. + +Whether a managed platform allows the opclass is **per-platform**, decided by +whether its admin role passes (or the vendor's engine delegates) that +superuser check: + +| Platform | ORE opclass installable? | Source | +| --- | --- | --- | +| Self-hosted / containerised PostgreSQL (superuser) | ✅ | stock behaviour | +| Aurora PostgreSQL (13+, and 12.12.0+) | ✅ — delegated to `rds_superuser` | [Aurora release notes 12.12.0](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/AuroraPostgreSQL.Updates.html): "Added support for the rds_superuser role to execute CREATE OPERATOR CLASS…" | +| RDS for PostgreSQL (non-Aurora) | ✅ — the master user can | verified in production CipherStash deployments running ORE with the custom operator class on RDS (AWS does not release-note the delegation the way it does for Aurora — confirm on your instance with the probe below if in doubt) | +| Cloud-hosted Supabase | ❌ — the `postgres` role cannot | [supabase/supautils#72](https://github.com/supabase/supautils/issues/72) (open feature request; "must be superuser to create an operator family") | +| Google Cloud SQL | ❌ | ["Unsupported features"](https://cloud.google.com/sql/docs/postgres/features): "Any feature that requires SUPERUSER" (opclasses are not among the exceptions) | +| Azure Database for PostgreSQL Flexible Server | ❌ (admin is `NOSUPERUSER`; no documented carve-out) | [Azure security docs](https://learn.microsoft.com/en-us/azure/postgresql/security/security-access-control) | + +To check any role/platform directly: + +```sql +DO $$ BEGIN + CREATE OPERATOR FAMILY _priv_probe USING btree; + DROP OPERATOR FAMILY _priv_probe USING btree; + RAISE NOTICE 'this role can create operator classes/families'; +EXCEPTION WHEN insufficient_privilege THEN + RAISE NOTICE 'this role cannot create operator classes/families (SQLSTATE 42501)'; +END $$; +``` + +> **This is a different mechanism from the extension restriction.** Custom +> *extensions* are impossible on managed platforms because `CREATE EXTENSION` +> needs the extension's files on a filesystem customers cannot touch — vendors +> ship a vetted set and may restrict it further (e.g. RDS's +> `rds.allowed_extensions`). The EQL opclass ships **no files**: it is pure +> catalog DDL whose support functions ride on pgcrypto (on every platform's +> supported/trusted list). Only the superuser gate above decides it — +> extension allow-lists are never the reason. + +If a database was **installed with the opclass and later loses it** (or an +`_ord_ore` column otherwise exists without it), note that `CREATE INDEX … +btree (eql_v3.ord_term_ore(col))` still succeeds by silently binding +`record_ops` — an index that never engages. See [the detection +query](./database-indexes.md#range-queries-and-order-by) in the indexes guide. + ## Granting access (the opt-in step) For an application role (for example a Supabase `authenticated` / `anon` role, or @@ -36,7 +97,8 @@ GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3_internal TO app_role; -- pgcrypto (Supabase installs it here). The ORE comparison behind ordering and -- MIN/MAX calls pgcrypto `encrypt()`, so ordered/aggregated queries need USAGE --- on its schema. If pgcrypto lives elsewhere, grant USAGE on that schema instead. +-- on its schema. (The installer accepts pgcrypto only in `extensions` or in +-- `public`; `public` needs no grant.) GRANT USAGE ON SCHEMA extensions TO app_role; -- Optionally keep future functions granted: @@ -58,8 +120,7 @@ needs `USAGE` on it anyway. The exact requirement is path-dependent: | Ordering (`<` `<=` `>` `>=` / `eql_v3.lt`…) | ✅ | ✅ | only on `_ord_ore` / `text_search_ore` | | `MIN` / `MAX` aggregates | ✅ | ✅ | only on `_ord_ore` / `text_search_ore` | | jsonb containment read (`@>` `<@` / `ste_vec_contains`) | ✅ | — | — | -| Cast/write raw JSON → `public.eql_v3_json_search` | ✅ | ✅ | — | -| Cast/write raw JSON → a scalar domain (`public.eql_v3_integer`…) | ✅ | — | — | +| Cast/write raw JSON → `public.eql_v3_json_search` or a scalar domain (`public.eql_v3_integer`…) | — | — | — | | Cast a query operand → `eql_v3.query_` / `eql_v3.query_json` | ✅ | — | — | Why the internal grant is needed even though you only call public objects: @@ -79,15 +140,19 @@ Why the internal grant is needed even though you only call public objects: installer places in the `extensions` schema — hence the `USAGE` there. The CLLW-OPE variants (`_ord`, `_ord_ope`, `text_search`) compare native `bytea` and need no `extensions` grant. -- **Casting raw jsonb to `public.eql_v3_json_search` or `eql_v3.query_json`** fires a - domain `CHECK` that calls an `eql_v3_internal.is_valid_*` validator. (Scalar - domain CHECKs — and, since issue #354, the `public.eql_v3_json_entry` CHECK — are - pure structural jsonb tests, so casting to those domains needs no internal - grant.) +- **Casting raw jsonb to `public.eql_v3_json_search`** fires a domain `CHECK` + that calls the `public.eql_v3_is_valid_ste_vec_*` validators — deliberately + kept in `public` so application table columns survive an EQL schema + uninstall — so it needs **no** schema grant at all. (Scalar domain CHECKs — + and, since issue #354, the `public.eql_v3_json_entry` CHECK — are pure + structural jsonb tests, likewise grant-free. Casting to the + `eql_v3.query_*` operand domains needs `USAGE` on `eql_v3` only because the + domains themselves live there.) The hand-written jsonb containment **read** path (`eql_v3.ste_vec_contains` and -the `@>` / `<@` operators over it) is `plpgsql` — never inlined — so it runs under -the public `eql_v3` grant alone. +the `@>` / `<@` operators over it) stays within `eql_v3` / `public` — the array +variant is `plpgsql` (never inlined) and the typed variant inlines only `eql_v3` +calls — so it runs under the public `eql_v3` grant alone. This behaviour is gated by `tests/sqlx/tests/v3_privilege_tests.rs`. So the public **function equivalents** (below) change *how* you invoke a supported diff --git a/docs/reference/query-performance.md b/docs/reference/query-performance.md index 7b4c6a561..303bfea39 100644 --- a/docs/reference/query-performance.md +++ b/docs/reference/query-performance.md @@ -7,7 +7,7 @@ Getting EQL-encrypted queries competitive with plain PostgreSQL comes down to on - [Creating indexes](./database-indexes.md#creating-indexes) — the `eql_v3.eq_term` / `ord_term` / `match_term` recipes (no operator class on a column). - [How index engagement works](./database-indexes.md#how-index-engagement-works) — extractor inlining and structural matching. - [Range queries and the `ORDER BY` sort-key trap](./database-indexes.md#range-queries-and-order-by) — write `ORDER BY` against the column's ordering extractor (`eql_v3.ord_term(col)` on `_ord`) to avoid a Sort node. -- [`GROUP BY` / `DISTINCT`](./database-indexes.md#group-by--distinct) — group on `eql_v3.eq_term(col)`, not the raw column, to stay inside `work_mem`. +- [`GROUP BY` / `DISTINCT`](./database-indexes.md#group-by--distinct) — group on the column's extractor, not the raw column, to stay inside `work_mem` (`eql_v3.eq_term(col)` on `hm`-carrying domains; `eql_v3.ord_term(col)` on the numeric-and-time ordering domains, which have no `eq_term`). - [GIN indexes for JSONB containment](./database-indexes.md#gin-indexes-for-jsonb-containment) — `public.eql_v3_json_search` document search. - [Building indexes on large tables](./database-indexes.md#performance-building-indexes-on-large-tables) — `maintenance_work_mem`, btree-vs-hash build scaling, the de-TOAST floor. - [Diagnosing queries with `EXPLAIN`](./database-indexes.md#diagnosing-queries-with-explain). diff --git a/docs/reference/sql-support.md b/docs/reference/sql-support.md index 5a3cd947b..93cbe5493 100644 --- a/docs/reference/sql-support.md +++ b/docs/reference/sql-support.md @@ -15,9 +15,9 @@ The capability of a column is fixed by the **domain variant you type it as**. Th Each scalar type `` is a family of `jsonb`-backed domains in `public`. The catalog scalar tokens that ship today are: -`smallint`, `integer`, `bigint`, `numeric`, `real`, `double`, `date`, `timestamp`, `text`, `boolean`. +`smallint`, `integer`, `bigint`, `numeric`, `real`, `double`, `date`, `timestamp`, `text`, `boolean`, `json` (the bare `public.eql_v3_json` scalar domain is storage-only, like `boolean` — the *queryable* JSON document domain is `public.eql_v3_json_search`, [below](#publiceql_v3_json_search-structured-encryption-for-json)). -(See [Adding a Scalar Encrypted-Domain Type](./adding-a-scalar-encrypted-domain-type.md) for how the family is generated.) The domains live in the `public` schema, so they survive `DROP SCHEMA eql_v3 CASCADE` — dropping `eql_v3` removes the operators, extractors, and aggregates but leaves the `public`-typed columns and their data intact. Their extracted index-term types are the self-contained `eql_v3_internal` SEM types (`eql_v3_internal.hmac_256`, `eql_v3_internal.ore_block_256`, `eql_v3_internal.bloom_filter`). +(See [Adding a Scalar Encrypted-Domain Type](./adding-a-scalar-encrypted-domain-type.md) for how the family is generated.) The domains live in the `public` schema, so they survive `DROP SCHEMA eql_v3 CASCADE` — dropping `eql_v3` removes the query operators, extractors, and aggregates (the blockers, bound to `eql_v3_internal` functions, survive) but leaves the `public`-typed columns and their data intact. Their extracted index-term types are the self-contained `eql_v3_internal` SEM types (`eql_v3_internal.hmac_256`, `eql_v3_internal.ope_cllw`, `eql_v3_internal.ore_block_256`, `eql_v3_internal.bloom_filter`). Every scalar generates a storage-only variant plus the query variants its capabilities allow: @@ -35,14 +35,14 @@ Every scalar generates a storage-only variant plus the query variants its capabi Notes: -- The bare `public.` variant carries no index term and **blocks every comparison operator** — it is storage / decryption only. Type the column as `_eq` or `_ord` (or cast at the call site, e.g. `col::public.eql_v3_integer_ord`) when you need to query. +- The bare `public.` variant carries no index term and **blocks every comparison operator** — it is storage / decryption only. Type the column as `_eq` or `_ord` (or cast at the call site, e.g. `col::public.eql_v3_integer_ord` — this succeeds only if the stored payloads already carry the `op` term; otherwise the domain CHECK raises) when you need to query. - `_ord` and `_ord_ope` are **twins**: byte-identical surfaces backed by the CLLW-OPE term. `op` is a hex-encoded, order-preserving ciphertext compared by native bytea ordering after hex-decode (no custom comparison protocol, and `eql_v3_internal.ope_cllw` is a domain over `bytea`, so a functional btree on `eql_v3.ord_term(col)` uses the default operator class and needs no superuser). `_ord` is the recommended name; `_ord_ope` documents the scheme explicitly. -- `_ord_ore` exposes the **same ordered surface** backed by the block-ORE term (`ob`) instead, compared by the custom N-block protocol. Use it when you specifically need block-ORE. Caveat: its btree operator class is created by a superuser-only `DO` block that is **silently skipped** without that privilege. When it is missing, `CREATE INDEX … btree (eql_v3.ord_term_ore(col))` still *succeeds* — PostgreSQL falls back to `record_ops` on the composite — but that opfamily does not contain the ORE comparison operators, so the index never engages and the ordering it stores is not the ORE ordering. Verify with `\d+` that the index opclass is `ore_block_256_operator_class`, not `record_ops`. -- On `text_ord` / `text_ord_ope` / `text_ord_ore`, `=` / `<>` route through `hm` (exact HMAC) — ordering terms over text are not equality-lossless. +- `_ord_ore` exposes the **same ordered surface** backed by the block-ORE term (`ob`) instead, compared by the custom N-block protocol. Use it when you specifically need block-ORE. Caveat: its btree operator class is created by a superuser-gated `DO` block that is **skipped (with only a `NOTICE`)** without that privilege. When it is missing, `CREATE INDEX … btree (eql_v3.ord_term_ore(col))` still *succeeds* — PostgreSQL falls back to `record_ops` on the composite — but that opfamily does not contain the ORE comparison operators, so the index never engages and the ordering it stores is not the ORE ordering. Verify with `\d+` that the index opclass is `ore_block_256_operator_class`, not `record_ops`. +- For ` = text`, the ordering variants carry **more** than the table's generic cells say: `text_ord` / `text_ord_ope` carry `hm` **+** `op`, and `text_ord_ore` carries `hm` **+** `ob`, each with an `eql_v3.eq_term(col)` extractor alongside the ordering one. `=` / `<>` route through `hm` (exact HMAC) — ordering terms over text are not equality-lossless, unlike the numeric-and-time types where the injective ordering term serves equality directly. - `text_ord` accepts the empty string (its `op` term is well-formed and sorts first). `text_ord_ore` **rejects** it: encrypting `""` yields an empty ORE term (`ob: []`) that the domain CHECK refuses. - `=` / `<>` is the only searchable surface for `_eq`. On `_ord` variants the equality operators are available too (alongside the ordered ones). - `boolean` is **storage-only** by design — a two-value column has too little cardinality for any searchable index to be safe, so it ships only `public.eql_v3_boolean` (no `_eq` / `_ord`). -- `LIKE` / `ILIKE` (`~~` / `~~*`) and the native JSONB operators are **blocked on every scalar domain variant** — they are meaningless on a scalar payload. Text matching is the bloom-filter `@@` (`eql_v3.matches`) on `text_match`, not `LIKE`. +- `LIKE` / `ILIKE` (`~~` / `~~*`) do not work on any scalar domain variant — but unlike the other ❌ cells they fail at operator *resolution* (PostgreSQL's "operator does not exist"), since no `~~` blocker is defined, rather than with the EQL "operator not supported" exception. Text matching is the bloom-filter `@@` (`eql_v3.matches`) on `text_match`, not `LIKE`. - `MIN` / `MAX` are exposed only on the ordered variants, as `eql_v3.min(public._ord)` / `eql_v3.max(...)` (and likewise on `_ord_ope` / `_ord_ore`) — see [EQL Functions Reference](./eql-functions.md#eql_v3min--eql_v3max-per-domain). --- @@ -82,16 +82,17 @@ This matrix covers higher-level SQL constructs. As above, ✅ requires the colum | `WHERE col IN (…)` | desugars to `=` | `_eq`, `_ord`, `text_search`, `text_search_ore` | | `ORDER BY col` | meaningful only with an ordering term | `_ord`, `text_search`, `text_search_ore` | | `GROUP BY col` / `DISTINCT` | needs an equality term | `_eq`, `_ord`, `text_search`, `text_search_ore` | -| `MIN(col)` / `MAX(col)` | `eql_v3.min(public._ord)` / `max` — type the column as `_ord` or cast at the call site (`eql_v3.min(col::public.eql_v3_integer_ord)`) | `_ord` | +| `MIN(col)` / `MAX(col)` | `eql_v3.min(public._ord)` / `max` — type the column as `_ord` or cast at the call site (`eql_v3.min(col::public.eql_v3_integer_ord)`) | `_ord`, `text_search`, `text_search_ore` | | `COUNT(col)` / `COUNT(DISTINCT col)` | plain `COUNT(col)` needs no term; `DISTINCT` needs an equality term | any / `_eq` for `DISTINCT` | -| `JOIN … ON lhs.col = rhs.col` | both sides must share the same keyset and a matching variant | `_eq`, `_ord`, `text_search` | +| `JOIN … ON lhs.col = rhs.col` | both sides must share the same keyset and a matching variant | `_eq`, `_ord`, `text_search`, `text_search_ore` | Notes: +- **`_ord` in the "requires" column is shorthand for all three ordering variants** — `_ord`, its explicit twin `_ord_ope`, and the block-ORE `_ord_ore`. Every row that lists `_ord` holds for all three (they share the operator surface, and `eql_v3.min` / `max` are generated for each); `_ord_ope` and `_ord_ore` are spelled out in the domain-variant table above. - **Cross-column / cross-table comparisons** (joins, `IN (subquery)`, set-operation dedup) require both sides to have been encrypted with the *same* keyset and a matching variant. - **`ORDER BY`** without an ordering term will not produce a meaningful order — type the column as an `_ord` variant when ordering matters. - **Aggregates beyond `MIN` / `MAX`** (`SUM`, `AVG`, …) are not supported on encrypted values — decrypt at the application boundary and aggregate client-side. -- **Parameter binding**: CipherStash Proxy rewrites bound parameters so the encrypted operator and any functional indexes are selected. When bypassing the proxy, type the parameter (`$1::public.eql_v3_integer_ord`) so the encrypted operator resolves rather than the native `jsonb` one. +- **Parameter binding**: [CipherStash Proxy](https://github.com/cipherstash/proxy) rewrites bound parameters so the encrypted operator and any functional indexes are selected. When bypassing the proxy, type the parameter with the matching **query-operand domain** — `$1::eql_v3.query_integer_ord`, `$1::eql_v3.query_text_eq`, … — so the encrypted operator resolves rather than the native `jsonb` one. Every queryable variant has an `eql_v3.query__` twin, and query payloads must use it: they are term-only (no ciphertext), so casting one to the *storage* domain (`public.eql_v3_integer_ord`) fails its CHECK, and a bare uncast literal is ambiguous between the query and `jsonb` overloads and will not resolve. --- @@ -148,7 +149,7 @@ Exact equality is always value-selector presence (`@>`), injective for every typ | extracted-leaf `<` `<=` `>` `>=` | `eql_v3.ord_term(public.eql_v3_json_entry)` | ordered comparison on an extracted String / Number leaf. | | `MIN` / `MAX` of extracted leaf | `eql_v3.min(public.eql_v3_json_entry)` / `max` | over an extracted ordered leaf. | | `eql_v3.jsonb_path_query(doc, sel)` | path query | set-returning; yields encrypted entries. Also `jsonb_path_query_first`, `jsonb_path_exists`. | -| `eql_v3.jsonb_array_length/elements/elements_text(doc)` | array helpers | length / set-returning elements / element text. | +| `eql_v3.jsonb_array_length/elements(doc)` | array helpers | length / set-returning encrypted elements. (`jsonb_array_elements_text` was removed in 3.0 — a bare-ciphertext stream is no longer independently decryptable.) | > **Typed operands (important).** The selector / needle operand must carry a **known type** — a typed parameter (`$1`, which the Proxy supplies) or an explicit cast (`doc -> 'sel'::text`, `$1::eql_v3.query_json`). A bare untyped literal (`doc -> 'sel'`) resolves to the **native `jsonb` operator** (PostgreSQL reduces the `public.eql_v3_json_search` domain to its `jsonb` base type for an unknown-typed RHS) and silently returns native jsonb semantics instead of the encrypted operator. diff --git a/docs/tutorials/proxy-configuration.md b/docs/tutorials/proxy-configuration.md index 93f003916..49ed93781 100644 --- a/docs/tutorials/proxy-configuration.md +++ b/docs/tutorials/proxy-configuration.md @@ -38,7 +38,7 @@ ALTER TABLE users ADD COLUMN encrypted_name public.eql_v3_text_match; ALTER TABLE users ADD COLUMN encrypted_profile public.eql_v3_json_search; ``` -The variant fixes the column's searchable surface: `_eq` for `=`, `_ord` for ordering/range, `text_match` for `@>` token containment, `public.eql_v3_json_search` for encrypted JSON. The bare `eql_v3.` variant is storage/decryption only. +The variant fixes the column's searchable surface: `_eq` for `=`, `_ord` for ordering/range, `text_match` for `@@` token matching, `public.eql_v3_json_search` for encrypted JSON. The bare `public.eql_v3_` variant is storage/decryption only. ## 2. Configure searchable encryption in the client @@ -47,7 +47,7 @@ Tell the encryption client which columns to encrypt and which index terms to emi - **CipherStash Stack** — define the columns and indexes in the schema. See the [CipherStash Stack schema reference](https://cipherstash.com/docs/stack/cipherstash/encryption/schema). - **CipherStash Proxy** — configure the encrypted columns in the Proxy's mapping config. See [CipherStash Proxy](https://github.com/cipherstash/proxy). -The terms the client emits (`hm` for equality, `ob` for ordering, `bf` for match, ste_vec for JSON) must match the column's domain variant from step 1 — e.g. configure an equality index for a column typed `public.eql_v3_text_eq`. +The terms the client emits (`hm` for equality, `op` for ordering — `ob` on the `_ord_ore` variants, `bf` for match, ste_vec for JSON) must match the column's domain variant from step 1 — e.g. configure an equality index for a column typed `public.eql_v3_text_eq`. ## 3. Create functional indexes @@ -67,7 +67,7 @@ Run writes and reads through CipherStash Proxy. On insert, the Proxy encrypts th ```sql -- Through the Proxy: the plaintext is encrypted on the way in INSERT INTO users (encrypted_email) -VALUES ('{"v":2,"k":"pt","p":"test@example.com","i":{"t":"users","c":"encrypted_email"}}'); +VALUES ('{"v":3,"k":"pt","p":"test@example.com","i":{"t":"users","c":"encrypted_email"}}'); -- Through the Proxy: the ciphertext is decrypted on the way out SELECT encrypted_email FROM users; @@ -84,7 +84,7 @@ Type the query operand (the Proxy supplies typed parameters automatically; in ha ```sql SELECT * FROM users WHERE encrypted_email = $1; -- operator-free form (e.g. Supabase): -SELECT * FROM users WHERE eql_v3.eq(encrypted_email, $1::public.eql_v3_text_eq); +SELECT * FROM users WHERE eql_v3.eq(encrypted_email, $1::eql_v3.query_text_eq); ``` **Range / ordering** (`public.eql_v3_timestamp_ord`): @@ -93,10 +93,10 @@ SELECT * FROM users WHERE eql_v3.eq(encrypted_email, $1::public.eql_v3_text_eq); SELECT * FROM events WHERE encrypted_at < $1 ORDER BY eql_v3.ord_term(encrypted_at) DESC; ``` -**Full-text match** (`public.eql_v3_text_match`) — bloom-filter token containment, not `LIKE`: +**Full-text match** (`public.eql_v3_text_match`) — bloom-filter token matching (`@@`), not `LIKE` (and not the containment operators, which raise on this domain): ```sql -SELECT * FROM users WHERE encrypted_name @> $1::public.eql_v3_text_match; +SELECT * FROM users WHERE encrypted_name @@ $1::eql_v3.query_text_match; ``` **Encrypted JSON** (`public.eql_v3_json_search`) — containment and field access; see [EQL with JSON and JSONB](../reference/json-support.md): @@ -118,7 +118,7 @@ SELECT encrypted_profile -> 'email_selector'::text FROM users; ## Troubleshooting -**Operator resolves to native `jsonb` / returns `NULL` instead of searching.** The query operand was an untyped literal, so PostgreSQL flattened the `eql_v3` domain to `jsonb`. Type the operand (`$1::public.eql_v3_text_eq`, `$1::eql_v3.query_json`) — the Proxy does this automatically. +**Operator resolves to native `jsonb` / returns `NULL` instead of searching.** The query operand was an untyped literal, so PostgreSQL flattened the `eql_v3` domain to `jsonb`. Type the operand with the matching query-operand domain (`$1::eql_v3.query_text_eq`, `$1::eql_v3.query_json`) — the Proxy does this automatically. **`=` returns no rows.** The column's values do not carry an `hm` equality term. Confirm the client is configured to emit the right term for the column's variant (step 2), and that data was written through the Proxy after configuring it. diff --git a/docs/upgrading/v2.3.md b/docs/upgrading/v2.3.md index 4853ef3cc..757fe5077 100644 --- a/docs/upgrading/v2.3.md +++ b/docs/upgrading/v2.3.md @@ -147,7 +147,7 @@ The `eql_v2.jsonb_path_query`, `jsonb_path_query_first`, and `jsonb_path_exists` -- inlines to a native `jsonb @>` over the same expression so -- the planner engages Bitmap Index Scan. CREATE INDEX users_data_stevec_query_idx - ON users USING gin (eql_v2.to_stevec_query(data_encrypted)::jsonb jsonb_path_ops); + ON users USING gin ((eql_v2.to_stevec_query(data_encrypted)::jsonb) jsonb_path_ops); -- Query shape: SELECT * FROM users @@ -156,7 +156,7 @@ The `eql_v2.jsonb_path_query`, `jsonb_path_query_first`, and `jsonb_path_exists` 4. **Update query recipes.** Where you used `eql_v2.jsonb_path_query_first(col, 'sel')` *only to extract a value for hash/equality*, switch to the typed chain — either `WHERE col -> 'sel' = $1::eql_v2.ste_vec_entry` (operator form; planner picks the inlined `eq_term` extractor) or directly `eql_v2.eq_term(col -> 'sel')` if you need the bytea term. `jsonb_path_query_first` remains the right tool when you actually want the encrypted value back (e.g. projection followed by decryption). -**Module organisation note.** The new function `eql_v2.eq_term(eql_v2.ste_vec_entry)` lives in `src/ste_vec/eq_term.sql`. The root-level `eql_v2.hmac_256(val)` and `eql_v2.hmac_256(val jsonb)` overloads remain in `src/hmac_256/functions.sql` and are now also inlinable SQL — see the U-002 amendment for the behaviour change. +**Module organisation note.** At the time of the 2.3 release, the new function `eql_v2.eq_term(eql_v2.ste_vec_entry)` lived in `src/ste_vec/eq_term.sql`, and the root-level `eql_v2.hmac_256(val)` / `eql_v2.hmac_256(val jsonb)` overloads in `src/hmac_256/functions.sql`, all inlinable SQL — see the U-002 amendment for the behaviour change. (Those v2 source paths were removed from this repository with the 3.0 release; `src/` now holds only the `v3/` tree.) **Behavioural change to be aware of.** Failure modes differ by operation when a column hasn't been re-encrypted (see [U-002](#u-002-equality-and-hashing-require-hmac) for the exact equality / hashing semantics): @@ -300,4 +300,4 @@ The application-level surface that's hardest to roll back is U-002's "raise inst ## See also - [`CHANGELOG.md`](../../CHANGELOG.md) — full enumeration of changes in 2.3. -- [Database indexes reference](../reference/database-indexes.md) — canonical recipe for functional indexes. +- [Database indexes reference](../reference/database-indexes.md) — canonical recipe for functional indexes. (That page now documents the `eql_v3` surface; the recipes here remain the v2 forms.) diff --git a/docs/upgrading/v3.0.md b/docs/upgrading/v3.0.md index 15e50dd57..f800d1fb0 100644 --- a/docs/upgrading/v3.0.md +++ b/docs/upgrading/v3.0.md @@ -1,15 +1,21 @@ # Upgrading to EQL 3.0 -`3.0.0` is a breaking release. This guide currently covers the `eql_v3` -envelope-version bump; the release owner extends it with notes for the other -pending breaking changes (notably the `eql_v2` schema removal) as the -release is prepared. +`3.0.0` is a breaking release. This guide covers the ten breaking changes in +the `eql_v3` surface (U-001 through U-010 below). + +> **Coming from EQL v2?** 3.0 also removes the `eql_v2` schema surface from +> this repository. There is no in-place conversion for `eql_v2_encrypted` +> columns — the v3 wire format and domains are different — so migrating an +> existing v2 deployment means re-encrypting each column into a v3 domain via +> the CipherStash client tooling, then dropping the v2 schema. See the +> [CipherStash Stack rollout tooling](https://github.com/cipherstash/stack) +> for the column-by-column migration path. ## TL;DR 1. **The `eql_v3` JSON envelope version is now `v: 3`** ([U-001](#u-001-eql_v3-payloads-carry-v-3)). Every `eql_v3` domain `CHECK` pins `VALUE->>'v' = '3'`, and the published payload bindings (Rust / TypeScript / JSON Schema) accept exactly `3`. Payloads carrying the legacy `v: 2` are rejected on insert or cast. 2. **Query-operand domains are `eql_v3.query_`** ([U-002](#u-002-query-operand-domains-are-eql_v3query_name)). Renamed from the `_query` suffix to a `query_` prefix AND moved from `public` into the `eql_v3` schema: `public.integer_eq_query` → `eql_v3.query_integer_eq`, and the encrypted-JSONB containment needle `public.jsonb_query` → `eql_v3.query_json`. Only affects 3.0.0 pre-release adopters — the old names never shipped in a final release. -3. **Non-superuser installs disable the ORE-backed domains** ([U-003](#u-003-non-superuser-installs-disable-the-ore-backed-domains)). When the installer role cannot create the ORE operator class (cloud-hosted Supabase, most managed Postgres), the `_ord_ore` / `text_search_ore` domains and their query twins now raise `feature_not_supported` on first use instead of silently degrading to seq scans. Use `_ord` (indexed OPE ordering), `_eq`, and `text_match` on those platforms. Superuser installs are unchanged. +3. **Non-superuser installs disable the ORE-backed domains** ([U-003](#u-003-non-superuser-installs-disable-the-ore-backed-domains)). When the installer role cannot create the ORE operator class (cloud-hosted Supabase, Cloud SQL, Azure Flexible Server — AWS RDS and Aurora *can*; see [Install privileges](../reference/permissions.md#install-privileges)), the `_ord_ore` / `text_search_ore` domains and their query twins now raise `feature_not_supported` on first use instead of silently degrading to seq scans. Use `_ord` (indexed OPE ordering), `_eq`, and `text_match` on those platforms. Superuser installs are unchanged. 4. **SteVec (encrypted JSONB) ordering is CLLW-OPE: path entries may carry `op`; entries never carry `hm` or `oc`** ([U-004](#u-004-stevec-ordering-terms-are-cllw-ope-op)). The `eql_v3_internal.ore_cllw` composite type, its custom comparator, operators, and superuser-only btree operator class are removed; entry ordering extracts `eql_v3.ord_term(entry)` — a bytea domain that orders under the DEFAULT btree opclass. Exact equality uses value-selector presence, not an entry term. Existing documents must be re-encrypted. 5. **The `_ord` domains are backed by CLLW-OPE, not block-ORE** ([U-005](#u-005-_ord-domains-are-backed-by-cllw-ope)). `public._ord` and `public.eql_v3_text_ord` now require the `op` term. The extractor keeps its name — `eql_v3.ord_term(col)` — but now returns `eql_v3_internal.ope_cllw`, so ordered functional indexes no longer need a superuser-created operator class. The block-ORE surface moves behind the qualified `eql_v3.ord_term_ore(col)` on `public._ord_ore`. Two behaviour changes ride along: `text_ord` now accepts the empty string, and `real`/`double` `_ord` columns now distinguish `-0.0` from `+0.0`. 6. **`public.eql_v3_text_search` is backed by CLLW-OPE too, and `public.eql_v3_text_search_ore` is new** ([U-006](#u-006-text_search-is-backed-by-cllw-ope)). The combined search domain now requires `hm` + `op` + `bf` and orders via `eql_v3.ord_term(col)`; equality (`hm`) and bloom fuzzy match (`bf`) are unchanged. The block-ORE shape (`hm` + `ob` + `bf`) lives on the new `public.eql_v3_text_search_ore`. `text_search` now accepts the empty string. @@ -136,7 +142,7 @@ the `schema/v3/*_query.json` JSON Schema files). ```sql -- Must succeed (the relocated operand domains exist): SELECT '{"v":3,"i":{},"hm":"aa"}'::jsonb::eql_v3.query_text_eq; -SELECT '{"sv":[{"s":"aa","hm":"bb"}]}'::jsonb::eql_v3.query_json; +SELECT '{"sv":[{"s":"aa"}]}'::jsonb::eql_v3.query_json; -- entries carry s (+ optional op) only; hm is rejected (U-004/U-010) -- Must fail with "type does not exist" (the pre-release names are gone): SELECT '{"v":3,"i":{},"hm":"aa"}'::jsonb::public.text_eq_query; SELECT '{"v":3,"i":{},"hm":"aa"}'::jsonb::public.query_text_eq; @@ -614,12 +620,12 @@ to the same array-containment the index supports: ```sql -- before: -SELECT * FROM t WHERE name @> $1::public.eql_v3_text_match; -SELECT * FROM t WHERE eql_v3.contains(name, $1::public.eql_v3_text_match); +SELECT * FROM t WHERE name @> $1::eql_v3.query_text_match; +SELECT * FROM t WHERE eql_v3.contains(name, $1::eql_v3.query_text_match); -- now: -SELECT * FROM t WHERE name @@ $1::public.eql_v3_text_match; -SELECT * FROM t WHERE eql_v3.matches(name, $1::public.eql_v3_text_match); +SELECT * FROM t WHERE name @@ $1::eql_v3.query_text_match; +SELECT * FROM t WHERE eql_v3.matches(name, $1::eql_v3.query_text_match); ``` Coordinate with the client: an SDK still emitting `eql_v3.contains(` for text match @@ -641,13 +647,19 @@ operator/function name moved — so no re-encryption or index rebuild is involve ### U-009: the uninstaller can exceed Postgres's default lock budget **What changed.** The v3 surface has grown to the point where the shipped -uninstaller — a single transaction whose core is `DROP SCHEMA eql_v3 CASCADE` -followed by `DROP SCHEMA eql_v3_internal CASCADE` — takes **6,433 locks** -(measured on this release: one per dropped function, operator, type, and cast, -all held until commit). PostgreSQL sizes its shared lock table once at startup -as `max_locks_per_transaction × max_connections`; at the defaults (64 × 100) -that is **6,400 slots for the whole cluster**, so a full uninstall now exceeds -what the default configuration nominally affords. +uninstaller — whose core is `DROP SCHEMA eql_v3 CASCADE` followed by +`DROP SCHEMA eql_v3_internal CASCADE` — takes **6,433 locks** when run as a +single transaction (measured on this release: one per dropped function, +operator, type, and cast, all held until commit). Note the shipped file +contains no `BEGIN`/`COMMIT` of its own: run it with +`psql --single-transaction -f cipherstash-encrypt-uninstall.sql` to get the +atomic behaviour this note describes — under plain autocommit each `DROP` +commits separately (fewer locks at once, but `eql_v3` can drop while +`eql_v3_internal` survives if the second statement fails). PostgreSQL sizes +its shared lock table once at startup as `max_locks_per_transaction × +(max_connections + max_prepared_transactions)`; at the defaults (64 × 100) +that is **6,400 slots for the whole cluster**, so a full single-transaction +uninstall now exceeds what the default configuration nominally affords. **Who is affected.** Anyone running `cipherstash-encrypt-uninstall.sql` on a cluster with the default `max_locks_per_transaction = 64` — in practice only @@ -662,9 +674,10 @@ ERROR: out of shared memory HINT: You might need to increase max_locks_per_transaction. ``` -Nothing is left half-dropped — the transaction aborts atomically — but the -error is confusing and the standard remedy needs a **server restart**, which is -worth knowing *before* scheduling an uninstall on a production cluster. +Nothing is left half-dropped — under `--single-transaction` the transaction +aborts atomically — but the error is confusing and the standard remedy needs a +**server restart**, which is worth knowing *before* scheduling an uninstall on +a production cluster. **What to do.** Either: