Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search

`src/v3/scalars/` holds the generated **encrypted-domain type families** — jsonb-backed PostgreSQL domains in the **`eql_v3` schema**, one domain per operator/index capability (`eql_v3.<T>` storage-only, `eql_v3.<T>_eq`, `eql_v3.<T>_ord`). The schema qualifier replaces the old version-prefixed name, so the domains are `eql_v3.int4`, `eql_v3.int4_eq`, `eql_v3.int4_ord`, `eql_v3.int4_ord_ore` — created in `eql_v3`, not `public`. Their extractors/wrappers/aggregates (`eql_v3.eq_term`, `eql_v3.ord_term`, `eql_v3.eq`/`lt`/…, `eql_v3.min`/`max`) also live in `eql_v3`, and the SEM index-term types they return and construct (`eql_v3.hmac_256`, `eql_v3.ore_block_256`) are **also `eql_v3`** — hand-written under `src/v3/sem/` so the whole v3 surface is self-contained (no `eql_v2.<symbol>` appears anywhere in v3 SQL; CI gates this via `mise run test:self_contained_v3` and the self-contained `release/cipherstash-encrypt.sql` installer). `eql_v3.int4` (PR #239, supersedes #225) is the reference scalar implementation; the catalog now generates a full surface for `int4`, `int2`, `int8`, `date`, `timestamptz`, `numeric`, `text`, `bool`, `float4`, and `float8`, all following this materializer pattern. `jsonb` is the only catalog scalar with no generated SQL surface yet — it needs a separate SQL design beyond the ordered-scalar materializer, and the `eql-domains` fixture catalog (`crates/eql-domains`) models its fixture values ahead of that surface.

Adding a scalar encrypted-domain type is one row in the Rust catalog `eql-domains::CATALOG` (`crates/eql-domains/src/lib.rs`): a `ScalarSpec` giving the type `token` (e.g. `int8`), its `ScalarKind` (the `kind` field), the `DomainSpec`s mapping each generated domain suffix to its fixed index `Term`s (`_eq => [Hm]`, `_ord`/`_ord_ore => [Ore]`), and the `Fixture` value list. Term capabilities are fixed in the `Term` enum's `impl` methods (with unit tests): `Hm` provides equality, and `Ore` provides equality plus ordering. There is no TOML manifest and no Python — the catalog is the source of truth, validated by the compiler (an undefined term or unknown scalar is a compile error) plus catalog `#[test]`s. `mise run build` runs `cargo run -p eql-codegen`, which regenerates the scalar SQL surface into `src/v3/scalars/<T>/` from `CATALOG` at the start of every build; that surface includes supported comparison wrappers plus blockers for native `jsonb` operators that would otherwise be reachable through domain fallback. `cargo run -p eql-codegen` regenerates every type at once (the same call `mise run build` uses; there is no per-type codegen task). The generated `*_types.sql` / `*_functions.sql` / `*_operators.sql` / `*_aggregates.sql` files are gitignored and never committed. The per-type plaintext fixture lists the SQLx matrix consumes are **not** a generated file — they are materialised from each `CATALOG` row at compile time as `eql_domains::INT4_VALUES` / `INT2_VALUES` (the `int_values!` macro) and read directly by `ScalarType::FIXTURE_VALUES`; a Rust source of truth no longer round-trips through a committed generated `.rs`. Generated SQL carries a `-- AUTOMATICALLY GENERATED FILE` header (the project-wide marker `docs:validate` greps on); change the catalog and rebuild, never hand-edit. Hand-written SQL beyond the fixed surface goes in `src/v3/scalars/<T>/<T>_extensions.sql` with no auto-generated header and explicit `-- REQUIRE:` edges — that file IS committed. `jsonb` is out of scope for this scalar materializer.
Adding a scalar encrypted-domain type is one row in the Rust catalog `eql-domains::CATALOG` (`crates/eql-domains/src/lib.rs`): a `DomainFamily` giving the type `name` (e.g. `int8`), its `ScalarKind` (the `kind` field), the `Domain`s mapping each generated (bare) domain name to its fixed index `Term`s (`eq => [Hm]`, `ord`/`ord_ore => [Ore]`), and the `Fixture` value list. Term capabilities are fixed in the `Term` enum's `impl` methods (with unit tests): `Hm` provides equality, and `Ore` provides equality plus ordering. There is no TOML manifest and no Python — the catalog is the source of truth, validated by the compiler (an undefined term or unknown scalar is a compile error) plus catalog `#[test]`s. `mise run build` runs `cargo run -p eql-codegen`, which regenerates the scalar SQL surface into `src/v3/scalars/<T>/` from `CATALOG` at the start of every build; that surface includes supported comparison wrappers plus blockers for native `jsonb` operators that would otherwise be reachable through domain fallback. `cargo run -p eql-codegen` regenerates every type at once (the same call `mise run build` uses; there is no per-type codegen task). The generated `*_types.sql` / `*_functions.sql` / `*_operators.sql` / `*_aggregates.sql` files are gitignored and never committed. The per-type plaintext fixture lists the SQLx matrix consumes are **not** a generated file — they are materialised from each `CATALOG` row at compile time as `eql_domains::INT4_VALUES` / `INT2_VALUES` (the `int_values!` macro) and read directly by `ScalarType::FIXTURE_VALUES`; a Rust source of truth no longer round-trips through a committed generated `.rs`. Generated SQL carries a `-- AUTOMATICALLY GENERATED FILE` header (the project-wide marker `docs:validate` greps on); change the catalog and rebuild, never hand-edit. Hand-written SQL beyond the fixed surface goes in `src/v3/scalars/<T>/<T>_extensions.sql` with no auto-generated header and explicit `-- REQUIRE:` edges — that file IS committed. `jsonb` is out of scope for this scalar materializer.

**Adding a new encrypted-domain type: follow `docs/reference/adding-a-scalar-encrypted-domain-type.md`.** The mechanics are fixed for ordered scalar domains; the catalog row only declares the token, kind, domain suffixes, and terms. New term behavior belongs in the `Term` enum's `impl` methods in `crates/eql-domains/src` with tests, not in free-form catalog data.
**Adding a new encrypted-domain type: follow `docs/reference/adding-a-scalar-encrypted-domain-type.md`.** The mechanics are fixed for ordered scalar domains; the catalog row only declares the name, kind, bare domain names, and terms. New term behavior belongs in the `Term` enum's `impl` methods in `crates/eql-domains/src` with tests, not in free-form catalog data.

Regeneration is deterministic: an identical `CATALOG` produces byte-identical SQL. If `mise run build` produces unexpected output, the change is in `crates/eql-domains/src` (the catalog/terms) or `crates/eql-codegen/src` (the renderers) — not in random run-to-run variation.

Expand Down
12 changes: 6 additions & 6 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,12 +181,12 @@ source of truth — the `CATALOG` const in
[`crates/eql-domains/src/lib.rs`](./crates/eql-domains/src/lib.rs). There is no
TOML manifest and no Python.

Each scalar type is one `ScalarSpec` row in `CATALOG`, declaring:
Each scalar type is one `DomainFamily` row in `CATALOG`, declaring:

- the type `token` (e.g. `int8`),
- the type `name` (e.g. `int8`),
- its `ScalarKind` (the `kind` field),
- the `DomainSpec`s mapping each generated domain suffix to its fixed index
`Term`s (`_eq => [Hm]`, `_ord` / `_ord_ore => [Ore]`), and
- the `Domain`s mapping each generated (bare) domain name to its fixed index
`Term`s (`eq => [Hm]`, `ord` / `ord_ore => [Ore]`), and
- the plaintext `Fixture` value list the SQLx test matrix consumes.

`mise run build` invokes `cargo run -p eql-codegen`, which regenerates the SQL
Expand Down Expand Up @@ -329,7 +329,7 @@ without a database at all.
### Adding a scalar encrypted-domain type

Adding a scalar encrypted-domain type (e.g. a new ordered numeric scalar) is one
`ScalarSpec` row in `eql-domains::CATALOG`
`DomainFamily` row in `eql-domains::CATALOG`
([`crates/eql-domains/src/lib.rs`](./crates/eql-domains/src/lib.rs)). New term
behaviour belongs in the `Term` enum's `impl` methods (with tests), not in
free-form catalog data. After editing the catalog, run `mise run build` to
Expand All @@ -338,7 +338,7 @@ regenerate the SQL surface.
Follow the reference guide:
[`docs/reference/adding-a-scalar-encrypted-domain-type.md`](./docs/reference/adding-a-scalar-encrypted-domain-type.md).
The mechanics are fixed for ordered scalar domains; the catalog row only
declares the token, kind, domain suffixes, and terms.
declares the name, kind, bare domain names, and terms.

A few footguns the generator exists to prevent — worth knowing when reading the
output:
Expand Down
2 changes: 1 addition & 1 deletion crates/eql-bindings/src/v3/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ pub trait DomainType {

/// Unqualified SQL domain name (e.g. `"int4_eq"`) — [`Self::sql_domain`]
/// minus the schema qualifier; matches `eql-domains`
/// `ScalarSpec::domain_name`.
/// `DomainFamily::domain_name`.
fn domain(&self) -> &'static str {
self.sql_domain()
.strip_prefix("eql_v3.")
Expand Down
16 changes: 8 additions & 8 deletions crates/eql-codegen/src/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

use crate::consts::*;
use crate::operator_surface::Operator;
use eql_domains::{DomainSpec, Term};
use eql_domains::{Domain, Term};

/// Build the minijinja environment with the embedded templates: one whole-file
/// template per output file (`types`/`functions`/`operators`/`aggregates`) plus
Expand Down Expand Up @@ -68,14 +68,14 @@ pub struct DomainBlock {

#[derive(serde::Serialize)]
pub struct TypesContext {
pub token: String,
pub family_name: String,
pub domains: Vec<DomainBlock>,
}

/// Build the per-domain block data (port of `render_domain_block`'s value logic,
/// minus comment prose and the CHECK skeleton — those are template-resident).
pub fn domain_block(token: &str, domain: &DomainSpec) -> DomainBlock {
let name = domain.name_with_token(token);
pub fn domain_block(family_name: &str, domain: &Domain) -> DomainBlock {
let name = domain.full_name(family_name);

let mut keys: Vec<String> = ENVELOPE_KEYS.iter().map(|k| sql_str(k)).collect();
for k in Term::term_json_keys(domain.terms) {
Expand Down Expand Up @@ -134,8 +134,8 @@ pub enum FnEntry {
#[derive(serde::Serialize)]
pub struct FunctionsContext {
pub requires: Vec<String>, // dependency paths only; template emits "-- REQUIRE:"
pub token: String,
pub name: String, // full domain name (token+suffix)
pub family_name: String,
pub name: String, // full domain name (family-name + "_" + domain-name)
pub dom: String, // schema-qualified domain, e.g. eql_v3.int4_eq
pub domain_lit: String, // sql_str(dom), defensively escaped for the RAISE literal
pub entries: Vec<FnEntry>,
Expand Down Expand Up @@ -209,7 +209,7 @@ pub struct OpEntry {
#[derive(serde::Serialize)]
pub struct OperatorsContext {
pub requires: Vec<String>,
pub token: String,
pub family_name: String,
pub name: String,
pub dom: String,
pub operators: Vec<OpEntry>,
Expand All @@ -236,7 +236,7 @@ pub fn operator_entry(op: &Operator, leftarg: &str, rightarg: &str, supported: b
#[derive(serde::Serialize)]
pub struct AggregatesContext {
pub requires: Vec<String>, // dependency paths only; template emits "-- REQUIRE:"
pub token: String,
pub family_name: String,
pub name: String,
pub dom: String, // schema-qualified domain, hoisted
pub aggregates: &'static [AggregateOp], // == AGGREGATE_OPS
Expand Down
41 changes: 32 additions & 9 deletions crates/eql-codegen/src/dump.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,14 @@ pub struct TypeEntry {

#[derive(Serialize)]
pub struct DomainEntry {
/// Test-name segment: the base domain (`suffix == ""`) is `storage`;
/// otherwise the suffix without its leading underscore (`_eq` → `eq`,
/// `_ord_ore` → `ord_ore`).
/// Test-name segment: the base domain (`name == ""`) is `storage`;
/// otherwise the bare domain name (`eq`, `ord`, …).
pub segment: String,
/// Raw catalog suffix (`""`, `_eq`, `_ord`, `_ord_ore`, `_match`).
pub suffix: &'static str,
/// The `suffix` wire field (`""`, `_eq`, `_ord`, `_ord_ore`, `_match`),
/// reconstructed by re-prefixing the bare domain name with `_` so the
/// emitted JSON stays byte-stable after the catalog dropped the leading
/// underscore from its stored domain names.
pub suffix: String,
/// SQL operators the domain's terms support, in catalog order. Empty for
/// the storage domain (no terms).
pub supported_ops: Vec<&'static str>,
Expand All @@ -45,17 +47,21 @@ pub fn dump_catalog() -> CatalogDump {
.domains
.iter()
.map(|d| DomainEntry {
segment: if d.suffix.is_empty() {
segment: if d.name.is_empty() {
"storage".to_string()
} else {
d.suffix.trim_start_matches('_').to_string()
d.name.to_string()
},
suffix: if d.name.is_empty() {
String::new()
} else {
format!("_{}", d.name)
},
suffix: d.suffix,
supported_ops: Term::operators_for_terms(d.terms),
})
.collect();
TypeEntry {
token: spec.token,
token: spec.name,
is_eq_only: spec.is_eq_only(),
domains,
}
Expand Down Expand Up @@ -95,6 +101,23 @@ mod tests {
assert_eq!(ord.supported_ops, ["=", "<>", "<", "<=", ">", ">="]);
}

/// Pins the hand-re-derived `suffix` wire field — the one channel with no
/// other automated reader — so its underscore-prefixed values stay
/// byte-stable after the catalog dropped the leading underscore from its
/// stored (now bare) domain names.
#[test]
fn int4_suffix_field_is_underscore_prefixed() {
let dump = dump_catalog();
let int4 = dump
.types
.iter()
.find(|t| t.token == "int4")
.expect("int4 present in catalog");

let suffixes: Vec<&str> = int4.domains.iter().map(|d| d.suffix.as_str()).collect();
assert_eq!(suffixes, ["", "_eq", "_ord_ore", "_ord"]);
}

#[test]
fn timestamptz_is_ordered() {
// timestamptz was promoted to the ordered shape once
Expand Down
Loading
Loading