Conversation
Contributor
|
Important Review skippedToo many files! This PR contains 863 files, which is 713 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (195)
📒 Files selected for processing (863)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Verification pass over docs/reference/adding-a-scalar-encrypted-domain-type.md found several stale or wrong claims; fix them all: - Drop the stale 'codegen gates the PostgreSQL test matrix' claim (sections 4 and 6); the gate was removed (test job needs only changes/setup/build-archive). - Fix filename tasks/pin_search_path.sql -> pin_search_path_v3.sql (5 refs). - Fix non-compiling example int_values!(.., INT4) -> INT4_FIXTURES. - Rewrite section 6 stage-4 to match generate_type: ensure_generated_paths_writable -> write_generated_file -> remove_generated_orphans (no clean_generated_files in that path). - Correct text domain count (five -> six) and 'operators' vs 'CREATE OPERATOR statements' wording; intro reference range 5-7 -> 5-8. - Misc precision nits: text pivots (only mid_pivot overridden), SignedScalar also for timestamptz, binding output paths, int_values! location, splinter constructors/comparators label, self-containment gate pointer.
docs: prune obsolete process docs, consolidate SQL doc guidance, fix scalar-domain reference
…ting
Un-ignore src/v3/scalars/<T>/*_{types,functions,operators,aggregates}.sql and
commit them in place, retiring the byte-identical tests/codegen/reference/
goldens. Both generated targets (SQL + Rust bindings) are now handled the same
way: committed in place, drift-gated by regenerate-and-`git diff`.
- .gitignore: drop the four generated-SQL globs (functions.sql / *_extensions.sql
stay committed as before).
- .gitattributes: mark the committed surface linguist-generated; drop the dead
golden rule.
- codegen:parity: rewrite as regenerate-in-place + `git diff --exit-code` +
untracked check (mirrors types:check for the bindings); delete
tests/codegen/reference/.
- parity tests: repoint at the committed src/v3/scalars/ tree
(committed_scalar_dirs_match_catalog_tokens,
every_generated_sql_file_starts_with_marker); drop the redundant
byte-comparison tests now covered by the git-diff gate.
- install-hooks + tasks/githooks/pre-commit: run codegen:parity + types:check
locally when a commit touches the catalog/generator/generated surfaces,
closing the local-staleness gap for both targets.
- docs: CLAUDE.md, DEVELOPMENT.md, docs/reference/* updated to the
committed-in-place model.
No user-facing change: release/cipherstash-encrypt.sql is byte-identical, so no
CHANGELOG entry.
- pre-commit hook: include staged deletions (-diff-filter=ACMRD) so the local drift gate also fires when a generated file is removed - docs: correct "CI reverts drift" -> "CI fails on drift" in CLAUDE.md and DEVELOPMENT.md; the gate fails on diff, it does not revert
The drift gate (and the sibling types:check) regenerates the generated surface in the working tree before diffing, rather than into a temp tree. Make that side effect explicit in the script header comment and both task descriptions so a developer running the gate ad hoc isn't surprised that it overwrites src/v3/scalars (as mise run build already does on every build).
chore: commit generated SQL surface in place; unify drift gating (eql_v3 / 3.0.0)
The order_by NULLS matrix arm built its temp plaintext column and typed NULL sentinel from PG_TYPE. That was fine while every scalar's domain token equalled its plaintext Postgres type, but timestamp's token is 'timestamp' while its plaintext is 'timestamp with time zone' (a UTC instant / DateTime<Utc>), so the column decoded as bare TIMESTAMP and sqlx rejected it. Add ScalarType::PLAINTEXT_SQL_TYPE (default = PG_TYPE) for the plaintext column type. The temporal impls override it, deriving the value from EqlPlaintext (the same ScalarKind-keyed source the fixture generator uses) so it cannot drift from a hand-typed literal. Kept on ScalarType rather than read from EqlPlaintext directly because some test scalars (JsonbEntryInt4, a view type) are ScalarTypes but deliberately not EqlPlaintexts; those inherit the PG_TYPE default. Makes PlaintextSqlType::as_str const so the derivation is a const initializer.
Two more temp-table builders (the index-engagement and index-scan/EXPLAIN arms) typed their write-only `plaintext` column from PG_TYPE. Harmless today — the column is never read back, so the implicit timestamptz->timestamp assignment cast never surfaces — but inconsistent with the nulls-ordering arm and a latent footgun if a future change reads the column. Use PLAINTEXT_SQL_TYPE in all three spots. Addresses review feedback. Also rustfmt: the timestamptz->timestamp rename shortened the CATALOG array below the width threshold, so rustfmt now collapses it to one line.
Set up release-plz to publish the eql-bindings crate to crates.io using crates.io Trusted Publishing (OIDC) — no long-lived token stored in the repo. - Add release-plz.toml scoping the release to eql-bindings only; mark every other workspace crate (eql-domains, eql-tests-macros, eql_tests) publish = false so release-plz and cargo publish can only ever touch eql-bindings. - Add .github/workflows/release-plz.yml: a `release` job (OIDC -> short-lived crates.io token via rust-lang/crates-io-auth-action, then release-plz release) and a `release-pr` job, both on push to main. - Add required crates.io metadata to eql-bindings/Cargo.toml (license = MIT, repository, homepage, readme, keywords, categories) — publish fails without a license. - Add a `cargo publish --dry-run` packaging gate to the rust-crates CI job so publish-blockers surface on PRs, not at release time. - Exclude the eql-bindings-v* crate tag from the three eql-keyed SQL-surface workflows (release-eql, release-postgres-eql-image, rebuild-docs) so a crate release never triggers an SQL build, Docker image, or docs-site rebuild. - Seed crates/eql-bindings/CHANGELOG.md and document the two release tag families in the workflows README. CIP-3224
…racle Capture the name-vs-type rationale in the test crate: EQL data is stored as jsonb, so the concrete Postgres temporal type appears only in the plaintext oracle columns the tests build. Document that PG_TYPE is the domain identifier while PLAINTEXT_SQL_TYPE is the real storage type, why they diverge only for timestamp (instant / DateTime<Utc> = timestamp with time zone vs the SQL-standard tz-naive timestamp), and why the oracle deliberately stays tz-aware rather than becoming a bare timestamp (would force NaiveDateTime + reintroduce session-TZ dependence).
Reconcile the eql-bindings release-plz setup with the canonical cipherstash-suite pattern (the org reference; see its RELEASING.md and commit 02a989405): - Run `release` before `release-pr` (needs: release). The previous parallel jobs would reproduce the recursive-release-PR bug the suite documents: release-pr running first sees the just-merged `chore: release` commit as unreleased and opens another PR. - Use release-plz's native crates.io Trusted Publishing (id-token: write) instead of rust-lang/crates-io-auth-action. No CARGO_REGISTRY_TOKEN; GitHub ops use the default GITHUB_TOKEN. - Sign release commits and tags with GPG via crazy-max/ghaction-import-gpg (GPG_PRIVATE_KEY secret), matching the suite. - Add cliff.toml (git-cliff changelog template, conventional-commit parsers, `review` skip) and point release-plz.toml at it; align the workspace block (git_tag_enable, git_release_enable, publish, semver_check = false). - Match the suite's generated crate CHANGELOG.md header. - Static concurrency group + workflow-level permissions. EQL-local conventions kept deliberately (consistent within EQL, not the suite): blacksmith runner and mise toolchain, and the eql-bindings-v* tag exclusions on the SQL-surface workflows. CIP-3224
Rename eql_v3 `timestamptz` scalar type to `timestamp`
…indings-crate ci(release): publish eql-bindings to crates.io via release-plz
…ning, cache lint coverage Addresses the remaining items from the changes-requested review (#374 (review)): - DEV-placeholder publish guards (finding 6): release-plz.yml refuses to publish when the committed crates/eql-bindings/sql bundle wasn't prepared for the crate's version (enforced only when the version isn't already on crates.io, so routine no-op runs still pass); the npm package gains a prepublishOnly gate (scripts/verify-release-assets.mjs) that covers BOTH publish paths — changeset publish (production) and npm-publish.mjs (prerelease) shell out to npm publish, which runs it. - Prerelease marker (finding 7): classify accepts exactly 'chore(release):' (a bare 'release:' doc commit can no longer trigger a publish) and short-circuits to skip when the identity's eql-typescript-v tag already exists. - One-identity-one-commit (finding 8): prerelease-publish-rust dispatches release-plz.yml against the eql-<V> tag (same pattern as build-image), not the branch, so the crate publishes from the exact release commit. - Supply chain (findings 9/10): npm pinned to an exact version in both publish jobs; mise-action cache disabled in release.yml and release-plz.yml and the no-workflow-caching lint extended to cover jdx/mise-action (with release-plz.yml added to its default file set, plus test cases). - Changelog ownership (finding 5): CHANGELOG.md header, CLAUDE.md, and releasing.md now say what is true — Changesets maintains packages/eql/CHANGELOG.md (the lockstep release changelog); the root file is the frozen pre-3.0 archive. - Access-model note (finding 12): releasing.md documents that prerelease publishes are gated by eql_v3 push access + branch protection. - Tests (coverage finding): bumpCargoPackageVersion extracted from sync-lockstep-versions.mjs, now section-anchored (can never rewrite a column-0 version line outside [package]) with vitest coverage; prepare-bindings-assets.test.sh covers the version-validation gate (wired into lint-release.yml). - Nits: no defaults.run.shell 'bash {0}' override in release.yml / release-plz.yml (GitHub's default bash is -eo pipefail); unused repoRoot dropped from copy-assets.mjs.
feat(release): publish EQL v3 language bindings (Rust + TypeScript) with bundled SQL
Lockstep prerelease pin via 'pnpm run version' (changesets pre-mode): package.json, Cargo.toml, and the bundled exact-version SQL assets all carry 3.0.0-alpha.3. Pushing this marker commit to eql_v3 triggers release.yml's prerelease path: SQL + docs GitHub release (eql-3.0.0-alpha.3), npm publish (@cipherstash/eql, dist-tag alpha, tag eql-typescript-v3.0.0-alpha.3), and the dispatched crate publish (eql-bindings-v3.0.0-alpha.3).
First attempt (b5c586f) failed on two jobs: - prerelease-publish-rust: 'gh workflow run' resolves the workflow via a local git repo, and the job has no checkout — fixed with --repo. - prerelease-publish-npm: ENEEDAUTH — npm Trusted Publishing is not yet configured for @cipherstash/eql + release.yml on npmjs.com (alpha.2 was published manually). Needs the npmjs-side publisher config; will fail again on this retry, which is fine: the eql-typescript tag is only created after a successful npm publish, so classify keeps allowing retries for this identity until npm lands. The marker subject re-triggers the prerelease path for 3.0.0-alpha.3: build-sql/docs update the existing eql-3.0.0-alpha.3 release in place (action-gh-release updates by tag), and the fixed dispatch publishes the crate from the eql-3.0.0-alpha.3 tag.
…red) npm and crates.io OIDC trusted publishers are now configured (@cipherstash/eql -> release.yml; eql-bindings -> release-plz.yml) and release-plz.yml is registered on the default branch (PR #383). This marker retry completes the two package legs: the eql-typescript tag doesn't exist yet so classify proceeds; SQL/docs update in place.
…ranch dispatch) Second retry failed on both package legs for new reasons: - npm ENEEDAUTH persisted despite trusted publishing being configured: mise-action runs after the 'upgrade npm' step and prepends mise's node (from [tools]) to PATH — its bundled npm 10.x has no OIDC trusted publishing, so the upgraded npm 11.5.1 never ran. The upgrade step now runs AFTER mise-action in both publish jobs. - release-plz errored 'cannot determine current branch': dispatching against the eql-<V> tag gives it a detached HEAD, which it refuses. prerelease-publish-rust now pins a release/eql-<V> branch at the release commit and dispatches against that — same exact-commit guarantee, real branch for release-plz. classify proceeds for 3.0.0-alpha.3 (the eql-typescript tag is still uncreated); SQL/docs update in place.
npm publish and the crate publish both succeeded on the previous retry (@cipherstash/eql@3.0.0-alpha.3 with provenance; eql-bindings@3.0.0-alpha.3 via crates.io trusted publishing). The only remaining failure was the eql-typescript tag push: with persist-credentials: false the git extraheader hack didn't authenticate ('could not read Username'). Create the tag ref via 'gh api' instead. This retry converges: npm publish skips (already published), the tag is created, release-plz no-ops (version already on crates.io).
Until the 3.0.0 final ships, the alphas are the package's only release line, so a bare 'npm install @cipherstash/eql' should resolve to the newest alpha rather than whichever version last happened to hold latest (alpha.2 had it from a manual publish while alpha.3 sat under the alpha tag). PRE_GA_LATEST in npm-publish.mjs flips the policy back to channel dist-tags after GA. Policy per James, 2026-07-08.
…warning Doxygen auto-links the bare word `text` (it collides with a documented symbol) into a cross-reference — even inside inline code. In the jsonb selector operator's @warning, `` `col -> 'sel'::text` `` became `<computeroutput>col -> 'sel'<ref>text</ref></computeroutput>`: a link nested inside inline code. Rendered to Markdown that surfaces as an unbalanced `<tt>`, which fails the downstream MDX/docs build (cipherstash/docs was broken by eql-3.0.0-alpha.3's API.md at line 2153, `Expected a closing tag for <tt>`). It also drops the `::`, so the cast read as `'sel'text`. Prefix the two `text` occurrences with Doxygen's `%` no-autolink marker (`::%text`, `-> %text`). Verified with doxygen: the nested <ref> is gone and `col -> 'sel'::text` now renders as clean, balanced inline code.
…arse fix(docs): skip name-dropped CREATE FUNCTION mis-parses
…tolink fix(docs): suppress Doxygen auto-linking of `text` in jsonb selector @warning
…talls (CIP-3468) CREATE OPERATOR CLASS requires superuser, so on cloud-hosted Supabase and most managed Postgres the installer has always attempted the ORE operator class and skipped it on insufficient_privilege — but the ORE-carrying domains still installed half-working: </> comparisons ran as unindexable seq scans while CREATE INDEX ... (eql_v3.ord_term(col)) and bare ORDER BY failed with opaque Postgres errors. The installer now capability-detects the skip (checking pg_opclass after the attempt) and poisons all 38 ORE-carrying domains — _ord/_ord_ore on every ordered scalar, text_search, and their eql_v3.query_* twins — with an always-raising CHECK constraint: the first value cast or inserted (including NULL) raises feature_not_supported (SQLSTATE 0A000) naming the domain and the platform-supported alternatives (_ord_ope for indexed CLLW-OPE ordering, _eq for equality, text_match for pattern match). Superuser installs are unchanged: the operator class is created and nothing is poisoned. The fallback (src/v3/scalars/ore_fallback.sql) is generated from the catalog by eql-codegen via a new ore_fallback.sql.j2 template, so new ORE-carrying families are covered by construction. The poison function honours the encrypted-domain footguns: LANGUAGE plpgsql (never inlined, RAISE cannot be planned away) and not STRICT (NULLs cannot slip through). Integration tests run the shipped installer under SET ROLE to a NOSUPERUSER role and derive the poisoned/functional split from the same catalog.
…IP-3468) Addresses PR #388 review feedback (tobyhede + internal review). NOT VALID (blocking finding): ALTER DOMAIN ... ADD CONSTRAINT validates existing stored data, and the poison raises unconditionally — so re-running the installer over a database already holding ORE values (written under an earlier superuser install, before the installing role was demoted) aborted inside the DO block, for exactly the users the fallback exists to help. For domains, NOT VALID does not weaken enforcement (coercion applies every constraint regardless of validation status, NULL included); it skips the existing-data scan. Pinned by a codegen unit test and a new integration test (reinstall_over_ore_data) that runs the shipped installer as a superuser role, stores an ORE row, demotes the role, re-installs — old row stays readable, new writes raise 0A000 — and re-installs once more to pin non-superuser re-install idempotency. Also from review: - closing RAISE NOTICE no longer advertises "_match domains" generically (only text has one); count field dropped for entries|length - domain name now sql_str-escaped in the poison CHECK literal, matching the alternatives string - U-003: NOT VALID semantics, the text_search split collateral on superuser->managed migrations (equality/match lost too, re-encryption required), and a scope caveat that the SteVec ore_cllw path keeps silent degradation (follow-up CIP-3471) - payload examples use the wire-contract integer "v":3 (the bindings reject a JSON string; only the domain CHECK's ->> coercion accepted it)
…l-ore-types-on-cloud-hosted-supabase feat(install): disable ORE-backed domains loudly on non-superuser installs (CIP-3468)
Public-schema encrypted-domain type names now carry an eql_v3_ version prefix (public.eql_v3_integer, public.eql_v3_text_eq, public.eql_v3_json, ...). The prefix stops EQL domains shadowing PostgreSQL built-in type names and gives each EQL version a distinct column-type namespace so multiple versions can coexist at runtime (future version migrations). Query-operand domains (eql_v3.query_*) are unchanged — their schema already versions them. The rule lives in the catalog (eql_domains::PUBLIC_TYPNAME_PREFIX, Domain::sql_typname); codegen renders SQL + bindings through it, the hand-written jsonb (SteVec) surface and bindings are updated to match, and every generated surface (src/v3/scalars, eql-bindings Rust/TS/JSON schema, packages/eql) is regenerated — never hand-edited. dump-catalog now emits the installed typname so the docs manifest derives names from the same source.
SteVec entries now carry hm XOR op; entry ordering extracts
eql_v3.ord_ope_term(entry) — the eql_v3_internal.ope_cllw bytea domain the
scalar _ord_ope surface already uses, ordering under the DEFAULT btree
opclass with a fully inlinable comparison chain. Removes the
eql_v3_internal.ore_cllw composite, its per-byte comparator, six operators,
and the superuser-only operator class — the last EQL surface that could not
index on cloud-hosted Supabase / managed Postgres.
- eql-bindings: SteVecTerm::OpeCllw{op} replaces OreCllw{oc}; from_v2
rejects oc sv entries (UnconvertibleOreTerm) — ORE bytes misorder under
byte comparison, so re-encryption is the only conversion
- fixtures: SteVec generation pinned to SteVecMode::Compat with a
fail-closed oc->op remap shim until cipherstash-client emits sv-level op
- upgrade note U-003; changeset (major, folds into 3.0.0 pre-mode)
CIP-3469
…IP-3472) Sweep every non-generated consumer of the public domain names onto the prefixed spellings: the SQLx suite (matrix/property/ope/fixture sources, the clean-install and splinter scripts, the regenerated public-surface golden), the doc tree (README, SUPABASE, reference/tutorial pages), and the v3.0 upgrade guide (new U-003 note with the ALTER COLUMN recipe; historical pre-release names in U-002 kept unprefixed). Two places derived the eql_v3.query_<name> twin from the storage name and would have produced query_eql_v3_*: property.rs::query_cast and the ope fixture macro now strip PUBLIC_TYPNAME_PREFIX first — the twin joins query_ to the bare catalog name (the eql_v3 schema already versions query operands, so they carry no prefix).
…ar nit The changeset body is copied verbatim into packages/eql/CHANGELOG.md at release time, where a ../docs/... link resolves to the nonexistent packages/docs/... — use the root-relative anchored form the existing changesets use. Also restore the blank line before U-003's Verification block and fix 'an CLLW-OPE' in a test assertion message.
…tevec-support-from-cllw-ore-to-ope feat(jsonb)!: switch SteVec ordering from CLLW-ORE (oc) to CLLW-OPE (op)
…o CIP-3472 Conflicts resolved by the repo's regenerate-don't-merge rule for every generated surface, taking upstream's oc→op / ore_fallback structure for hand-written files and re-applying the eql_v3_ prefix on top. Upstream surfaces new since the branch point are brought onto the prefixed names: the ore_fallback renderer's poison targets (via domain_name), its codegen test and v3_ore_fallback_tests (via DomainFamily::domain_name), and the ope_cllw SEM doc. The upgrade note moves to U-005 (upstream took U-003/ U-004). Public-surface golden regenerated against a live install.
Cuts the EQL 3.0.0-alpha.4 lockstep release: npm `@cipherstash/eql`, crate `eql-bindings`, and the bundled SQL all ship at this identity. Carries CIP-3469 (#390): SteVec ordering moves from CLLW-ORE (`oc`) to CLLW-OPE (`op`). `SteVecTerm` becomes `{ hm } | { op }`, the `eql_v3_internal.ore_cllw` type and its superuser-only operator class are gone, and `from_v2` fails closed on `oc` entries with UnconvertibleOreTerm. Entry ordering now extracts `eql_v3.ord_ope_term(entry)` under native bytea comparison, so a plain functional btree index engages on managed Postgres. Unblocks protectjs-ffi#129, whose 7 remaining failures are all `FromV2(MissingTerm { key: "hm|oc" })` against alpha.3. Generated by `pnpm run version`: changesets computed alpha.4 (prerelease mode absorbs the major bump), sync-lockstep-versions.mjs propagated it to Cargo.toml, and release:prepare_bindings_assets rebuilt the exact-version SQL into both packages. Verified: bundled SQL carries 0 `ore_cllw` and 300 `ord_ope_term` references, crate and npm SQL are byte-identical, and all four version artifacts agree.
The CIP-3472 rename sweep rewrote the match arms of norm() in v3_jsonb_operator_surface_tests, but those arms match what Postgres' format_type() RETURNS, not names in our source. Two things changed at once: the domains gained the eql_v3_ prefix, and json stopped being a reserved word, so format_type no longer quotes it. Output is now bare eql_v3_json / eql_v3_jsonb_entry (schema omitted, public being on search_path), where the arms still looked for the quoted json and bare jsonb_entry. Every operand normalised to itself, so all four surface assertions saw unprefixed names and failed on 3 of 4 PG17 shards. Match the real output, and keep the bare query_jsonb arm so the assertions survive a search_path change that would unqualify it.
…blic-eql-v3-types-with-eql_v3_ feat(types)!: prefix all public EQL v3 types with `eql_v3_` (CIP-3472)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the
eql_v3release line intomainfor the EQL 3.0.0 GA release. (Originally opened as the long-running v3 integration tracking PR; description updated now that the branch is release-ready.)Linear: CIP-3465
What this brings to
maineql_v3/eql_v3_internalsurface — theeql_v2schema, its database-side config management, and the multi-variant build were removed in 3.0.0;release/cipherstash-encrypt.sqlis now a single self-contained installer built fromsrc/v3aloneinteger,smallint,bigint,date,timestamp,numeric,text,boolean,real,double) driven by theeql-domains::CATALOG+eql-codegenpipeline, with committed drift-gated SQL undersrc/v3/scalars/src/v3/jsonb/eql-bindings) and TypeScript (@cipherstash/eql) binding packages generated from the same catalog, both bundling the SQL installerrelease.yml, publish-onlyrelease-plz.yml), validated end-to-end by the3.0.0-alpha.xprereleases cut from this branchMerge conflict note
.github/workflows/release-plz.ymlis an add/add conflict:maincarries a registration-only dispatch stub (#383) whose own commit message says to replace it with the full workflow wheneql_v3merges. Resolve by taking theeql_v3side (the full publish workflow).After merge
maintriggersrelease.yml(classifies as no-op — not achore(release):marker commit; finals are owned by the changesets Version PR flow),release-plz.yml(no-op — the committed crate version is already published), andbench-eql.ymlmainby merging the changesets "Version Packages" PR perdocs/development/releasing.md