Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 19 additions & 61 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ Store encrypted data alongside your existing data:
- [Local development (fastest)](#local-development-fastest)
- [Install into an existing database](#install-into-an-existing-database)
- [dbdev](#dbdev)
- [EQL Components](#eql-components)
- [Release artifacts](#release-artifacts)
- [Database Permissions](#database-permissions)
- [Getting started](#getting-started)
- [Enable encrypted columns](#enable-encrypted-columns)
- [Encrypt configuration](#encrypt-configuration)
Expand Down Expand Up @@ -72,92 +75,47 @@ EQL installs the following components into the `eql_v3` schema:
| --------------------------------------------------- | ------------- | ------------------------------------------------------------------- |
| `eql_v3` | Schema | Holds EQL operators, term extractors, comparison wrappers, and aggregates |
| `public.<T>`, `public.<T>_eq`, `public.<T>_ord` | Domain types | Per-scalar encrypted columns (one family per scalar: `integer`, `text`, `timestamp`, …) |
| `public.eql_v3_json` | Domain type | Encrypted JSON (structured-encryption) documents |
| `public.eql_v3_json_search` | Domain type | Searchable encrypted JSON (structured-encryption) documents |
| `public.eql_v3_json` | Domain type | Encrypted JSON, storage-only (no query surface) |
| `eql_v3.eq_term` / `ord_term` / `match_term` | Functions | Index-term extractors for functional indexes |


### `eql_v3` Schema

The `eql_v3` schema holds the operators, term extractors, comparison wrappers, and `MIN` / `MAX` aggregates for the encrypted-domain types. The encrypted-domain types themselves live in `public` (see below), and the internal SEM index-term types live in `eql_v3_internal`.

Encrypted columns are typed as `public` domains (e.g. `public.eql_v3_text_eq`, `public.eql_v3_json`), and the searchable surface available on a column is fixed by its domain **variant** — there is no database-side configuration state. Which index terms a value carries is decided by the encryption client (CipherStash Stack / CipherStash Proxy).
Encrypted columns are typed as `public` domains (e.g. `public.eql_v3_text_eq`, `public.eql_v3_json_search`), and the searchable surface available on a column is fixed by its domain **variant** — there is no database-side configuration state. Storage-only variants (`public.eql_v3_text`, `public.eql_v3_json`, …) carry no query surface at all; the searchable JSON domain is `public.eql_v3_json_search`. Which index terms a value carries is decided by the encryption client (CipherStash Stack / CipherStash Proxy).

The domain types deliberately live in `public`, not `eql_v3`, so application tables survive an EQL uninstall: `DROP SCHEMA eql_v3 CASCADE` removes the operators, extractors, and aggregates but leaves the `public`-typed columns (and their data) intact. Re-running the install script is idempotent.
The domain types deliberately live in `public`, not `eql_v3`, so application tables survive an EQL uninstall: `DROP SCHEMA eql_v3 CASCADE` removes the operators, extractors, and aggregates but leaves the `public`-typed columns (and their data) intact. Re-running the install script is safe for columns and data — but note it begins with that same `DROP SCHEMA eql_v3 CASCADE`, which also **cascade-drops any functional indexes** built on the `eql_v3` extractors. After a re-install or upgrade, re-create your encrypted-column indexes and `ANALYZE` (see [Database Indexes](docs/reference/database-indexes.md)).


### Release artifacts

EQL v3 prereleases can ship three artifacts under one identity: the SQL + docs
EQL v3 releases ship three artifacts under one identity: the SQL + docs
GitHub release, the Rust `eql-bindings` crate, and the TypeScript
`@cipherstash/eql` npm package. The language packages bundle the exact SQL
installer they were generated against.


## Database Permissions

EQL requires specific database privileges to install and operate correctly. The permissions needed depend on your deployment pattern.
The canonical reference is **[EQL permissions & grants](docs/reference/permissions.md)** — install privileges (including the one superuser-gated component, the ORE operator class, and which managed platforms allow it), the per-query-path grant matrix, and the operator-free function equivalents. The short version:

### Default Permissions (Recommended)

For most use cases, grant the following permissions to the database user that will install and use EQL:
- **Installing** needs `CREATE` on the database plus `CREATE`/`USAGE` on `public` — no superuser, except for the optional block-ORE operator class. Altering the tables that receive encrypted columns requires table *ownership* (PostgreSQL has no grantable `ALTER` table privilege).
- **The installer issues no `GRANT`s.** Access to `eql_v3` / `eql_v3_internal` is strictly opt-in per role.
- **A runtime role** that queries encrypted columns typically needs:

```sql
-- Database-level permissions
GRANT CREATE ON DATABASE your_database TO your_eql_user;

-- Schema permissions
GRANT USAGE ON SCHEMA public TO your_eql_user;
GRANT CREATE ON SCHEMA public TO your_eql_user;

-- User table permissions (for encrypted column constraints)
GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_eql_user;
-- Or grant ALTER on specific tables that will have encrypted columns:
-- GRANT ALTER ON TABLE your_table TO your_eql_user;
```

**Why these permissions are needed:**

- **CREATE ON DATABASE**: Required to create the `eql_v3` schema, domain types, and functions during installation
- **CREATE ON SCHEMA public**: Required to add encrypted columns (typed as `public` domains) to tables in the public schema
- **ALTER on user tables**: encrypted-domain `CHECK` constraints are validated on the user tables

### Splitting Read and Write Access

A common production pattern separates setup/migration permissions from runtime permissions:

#### Setup/Migration User (Write Access)

Use during database migrations and EQL installation:

```sql
-- All default permissions above, plus:
GRANT CREATE ON DATABASE your_database TO your_migration_user;
GRANT CREATE ON SCHEMA public TO your_migration_user;
GRANT ALTER ON ALL TABLES IN SCHEMA public TO your_migration_user;
```

#### Runtime User (Read Access)

Use for application queries in production:

```sql
-- EQL schema usage (resolves the encrypted operators / extractors)
GRANT USAGE ON SCHEMA eql_v3 TO your_app_user;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3 TO your_app_user;

-- eql_v3_internal holds the implementation the public eql_v3 operators dispatch
-- into. Grant it the same way as eql_v3 — explicitly, per runtime role.
-- The public operators inline into eql_v3_internal, so query roles need it too:
GRANT USAGE ON SCHEMA eql_v3_internal TO your_app_user;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3_internal TO your_app_user;

-- User table access (normal application permissions)
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLE your_tables TO your_app_user;
```

**Migration Workflow:**
1. Use the migration user to install EQL and add encrypted columns
2. Use the runtime user for normal application operations
3. Schema changes (adding/removing encrypted columns) require the migration user
See [permissions.md](docs/reference/permissions.md) for the pgcrypto-schema grant the ORE variants additionally need, and for narrowing grants per query path.


### dbdev
Expand All @@ -173,7 +131,7 @@ Once EQL is installed in your PostgreSQL database, you can start using encrypted

### Enable encrypted columns

Define encrypted columns using a `public` encrypted-domain type. Type the column as the **variant** for the capability you need — `public.eql_v3_text_eq` for equality, `public.<T>_ord` for range/ordering, `public.eql_v3_text_match` for full-text, `public.eql_v3_json` for encrypted JSON. Each is stored as `jsonb` with a `CHECK` constraint that validates the encrypted payload.
Define encrypted columns using a `public` encrypted-domain type. Type the column as the **variant** for the capability you need — `public.eql_v3_text_eq` for equality, `public.<T>_ord` for range/ordering, `public.eql_v3_text_match` for full-text, `public.eql_v3_json_search` for searchable encrypted JSON (`public.eql_v3_json` is the storage-only flavour). Each is stored as `jsonb` with a `CHECK` constraint that validates the encrypted payload.

**Example:**

Expand Down Expand Up @@ -202,7 +160,7 @@ In order to enable searchable encryption, you will need to configure your Cipher

## Performance

Query latency for searchable-encryption operations stays low across data set sizes. The numbers below are query-only medians (no decryption) from a full benchmark run against EQL 2.3 on PostgreSQL 17, across four row-count tiers.
Query latency for searchable-encryption operations stays low across data set sizes. The numbers below are query-only medians (no decryption) from a full benchmark run against EQL 2.3 on PostgreSQL 17, across four row-count tiers. (The JSON rows predate the 3.0 ste_vec redesign — field equality is now value-selector containment — so treat them as indicative, not current.)

| Family | Scenario | 10k | 100k | 1M | 10M |
|---|---|--:|--:|--:|--:|
Expand Down Expand Up @@ -314,7 +272,7 @@ To upgrade to the latest version of EQL, you can simply run the install script a
```

> [!NOTE]
> The install script will not remove any existing configurations, so you can safely run it multiple times.
> Re-running the install script is safe for your columns and data (the domain types live in `public` and survive). It does, however, begin with `DROP SCHEMA eql_v3 CASCADE`, which cascade-drops any **functional indexes** built on the `eql_v3` extractors — re-create them and `ANALYZE` after every upgrade (see [Database Indexes](docs/reference/database-indexes.md)).

#### Using dbdev?

Expand All @@ -326,11 +284,11 @@ Follow the instructions in the [dbdev documentation](https://database.dev/cipher

**A query returns no rows / silently runs native `jsonb` semantics**
- **Cause**: the query operand was an untyped literal, so PostgreSQL flattened the `eql_v3` domain to its `jsonb` base type and resolved the native operator
- **Solution**: type the operand — `WHERE col = $1::public.eql_v3_text_eq` (CipherStash Proxy supplies typed parameters automatically)
- **Solution**: type the operand with the matching query-operand domain — `WHERE col = $1::eql_v3.query_text_eq` (CipherStash Proxy supplies typed parameters automatically)

**Error: "operator not supported" (raised)**
- **Cause**: the operator is blocked for the column's domain variant (e.g. `<` on an `_eq` column, or `LIKE` on any encrypted column)
- **Solution**: type the column as a variant that carries the required term (see the [SQL support matrix](docs/reference/sql-support.md)); use `@>` rather than `LIKE` for text match
- **Solution**: type the column as a variant that carries the required term (see the [SQL support matrix](docs/reference/sql-support.md)); use `@@` rather than `LIKE` for text match

**`=` returns no rows on a populated column**
- **Cause**: the column's values do not carry an `hm` equality term
Expand Down
5 changes: 3 additions & 2 deletions crates/eql-bindings/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,9 @@ Shared wire fields are reusable newtypes in
| Newtype | Wire key | Inner | Backs |
|---------|----------|-------|-------|
| `Ciphertext` | `c` | `String` | every domain (envelope) |
| `Hmac256` | `hm` | `String` | `_eq` domains |
| `OreBlock256` | `ob` | `Vec<String>` | `_ord` / `_ord_ore` domains |
| `Hmac256` | `hm` | `String` | `_eq` domains + the text ordering/search domains |
| `OpeCllw` | `op` | `String` | `_ord` / `_ord_ope` domains, `text_search` |
| `OreBlock256` | `ob` | `Vec<String>` | `_ord_ore` domains, `text_search_ore` |
| `BloomFilter` | `bf` | `Vec<i16>` (signed!) | `_match` domains |

Note "v3" names the SQL schema generation (`eql_v3.*`); the JSON envelope
Expand Down
6 changes: 3 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ This directory contains the documentation for the Encrypt Query Language (EQL).
## Reference

- [EQL Functions Reference](reference/eql-functions.md) - Complete API reference for all EQL functions
- [SQL support matrix](reference/sql-support.md) - Which SQL operators and features each encrypted index enables
- [Database Indexes for Encrypted Columns](reference/database-indexes.md) - PostgreSQL B-tree index creation and usage
- [SQL support matrix](reference/sql-support.md) - Which SQL operators and features each encrypted-domain variant enables
- [Database Indexes for Encrypted Columns](reference/database-indexes.md) - Index recipes (hash / btree / GIN), engagement rules, and build performance for encrypted columns
- [Writing fast queries against EQL columns](reference/query-performance.md) - Performance overview (points to Database Indexes)
- [Adding a Scalar Encrypted-Domain Type](reference/adding-a-scalar-encrypted-domain-type.md) - How the `eql_v3.<T>` domain families are generated
- [EQL with JSON and JSONB](reference/json-support.md)
- [EQL payload / wire format](../crates/eql-bindings/README.md) - Canonical wire types for the encrypted payload (envelope `v`/`i`/`c` and the `hm`/`ob`/`bf` index terms)
- [EQL payload / wire format](../crates/eql-bindings/README.md) - Canonical wire types for the encrypted payload (envelope `v`/`i`/`c` and the `hm`/`op`/`ob`/`bf` index terms)
- [Client-side index configuration](https://cipherstash.com/docs/stack/cipherstash/encryption/schema) - Configuring searchable encryption in CipherStash Stack / CipherStash Proxy

## Tutorials
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/WHY.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ EQL enables encryption in use without significant changes to your application co
A variety of searchable encryption techniques are available, including:

- **Matching** - Equality or partial matches
- **Ordering** - comparison operations using order revealing encryption
- **Ordering** - comparison operations using order-preserving encryption (CLLW-OPE by default; block-ORE as an opt-in variant)
- **Uniqueness** - enforcing unique constraints
- **Containment** - containment queries using structured encryption

Expand Down
25 changes: 15 additions & 10 deletions docs/development/sql-documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ prefix (not plain `--`). Coverage and required tags are checked by `mise run doc
--! bare-form queries.
--!
--! @param a public.eql_v3_integer_eq Encrypted value carrying an `hm` term
--! @return eql_v3.hmac_256 The equality index term
--! @return eql_v3_internal.hmac_256 The equality index term
--!
--! @example
--! CREATE INDEX ON users USING hash (eql_v3.eq_term(salary_eq));
--!
--! @see eql_v3.ord_term
CREATE FUNCTION eql_v3.eq_term(a public.eql_v3_integer_eq)
RETURNS eql_v3.hmac_256
RETURNS eql_v3_internal.hmac_256
AS $$ ... $$;
```

Expand Down Expand Up @@ -88,9 +88,9 @@ CREATE OPERATOR = (
```sql
--! @brief Encrypted-domain type for an equality-searchable integer column
--!
--! A `jsonb`-backed domain in the `eql_v3` schema. The `CHECK` requires the
--! A `jsonb`-backed domain in the `public` schema. The `CHECK` requires the
--! envelope keys (`v`, `i`, `c`), the equality term (`hm`), and pins the
--! payload version (`VALUE->>'v' = '2'`).
--! payload version (`VALUE->>'v' = '3'`).
--!
--! @see eql_v3.eq_term
CREATE DOMAIN public.eql_v3_integer_eq AS jsonb
Expand All @@ -104,7 +104,7 @@ CREATE DOMAIN public.eql_v3_integer_eq AS jsonb
--! @param $1 public.eql_v3_integer_ord Accumulated state
--! @param $2 public.eql_v3_integer_ord New value
--! @return public.eql_v3_integer_ord Updated state
CREATE FUNCTION eql_v3.min_sfunc(public.eql_v3_integer_ord, public.eql_v3_integer_ord)
CREATE FUNCTION eql_v3_internal.min_sfunc(public.eql_v3_integer_ord, public.eql_v3_integer_ord)
RETURNS public.eql_v3_integer_ord
AS $$ ... $$;

Expand All @@ -119,10 +119,12 @@ AS $$ ... $$;
--! @example
--! SELECT eql_v3.min(price_encrypted) FROM products;
--!
--! @see eql_v3.min_sfunc
--! @see eql_v3_internal.min_sfunc
CREATE AGGREGATE eql_v3.min(public.eql_v3_integer_ord) (
SFUNC = eql_v3.min_sfunc,
STYPE = public.eql_v3_integer_ord
SFUNC = eql_v3_internal.min_sfunc,
STYPE = public.eql_v3_integer_ord,
COMBINEFUNC = eql_v3_internal.min_sfunc,
PARALLEL = safe
);
```

Expand Down Expand Up @@ -186,8 +188,11 @@ CREATE FUNCTION eql_v3."[operator]"(...)
--!
--! @see eql_v3.eq_term
--! @note This is a transient type used only during query execution
--! @note Encrypted-domain types are jsonb-backed — always `AS jsonb`, never domain-over-domain
CREATE DOMAIN eql_v3.[type_name] AS jsonb;
--! @note SEM index-term types live in `eql_v3_internal` and are backed by whatever
--! base type the term needs (`hmac_256 AS text`, `ope_cllw AS bytea`) — never
--! domain-over-domain. (Encrypted-domain *column* types are the jsonb-backed
--! `public.eql_v3_*` domains.)
CREATE DOMAIN eql_v3_internal.[type_name] AS [base_type];
```

### Template: Aggregate Function
Expand Down
Loading
Loading